[GitHub] [flink] flinkbot edited a comment on pull request #14591: FLINK-20359 Added Owner Reference to Job Manager in native kubernetes
flinkbot edited a comment on pull request #14591: URL: https://github.com/apache/flink/pull/14591#issuecomment-756825239 ## CI report: * 35947579b06f3b4d67e151e2fc38565813076008 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12488) * 402056b79cfdcfeb7c718ca58ec2a96eed1554d0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12668) * 7100ec24b7b54983f13c1fe5a411aca18ce2e434 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20663) Managed memory may not be released in time when operators use managed memory frequently
[
https://issues.apache.org/jira/browse/FLINK-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young updated FLINK-20663:
---
Priority: Critical (was: Major)
> Managed memory may not be released in time when operators use managed memory
> frequently
> ---
>
> Key: FLINK-20663
> URL: https://issues.apache.org/jira/browse/FLINK-20663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Caizhi Weng
>Priority: Critical
> Fix For: 1.12.2
>
>
> Some batch operators (like sort merge join or hash aggregate) use managed
> memory frequently. When these operators are chained together and the cluster
> load is a bit heavy, it is very likely that the following exception occurs:
> {code:java}
> 2020-12-18 10:04:32
> java.lang.RuntimeException:
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate
> 512 pages
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:85)
> at
> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.(SimpleCollectingOutputView.java:49)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.(BytesHashMap.java:297)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:103)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:90)
> at LocalHashAggregateWithKeys$209161.open(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:506)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:834)
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209161.close(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.TableStreamOperator.dispose(TableStreamOperator.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:739)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:719)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:642)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:551)
> ... 3 more
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209766.close(Unknown
> Source)
> ... 8 more
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 512 pages
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:231)
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:83)
> ... 13 more
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 16777216 bytes, only 9961487 bytes are remaining. This usually
> indicates that you are requesting more memory than you have reserved.
> However, when running an old JVM version it can also be caused by slow
> garbage collection. Try to upgrade to Java 8u72 or higher if running on an
> old Java version.
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:164)
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:80)
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:229)
> ... 14 more
> {code}
> It seems that this is caused by relying on GC to release managed memory, as
> {{System.gc()}} may not trigger GC in time. See {{UnsafeMemoryBudget.java}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20663) Managed memory may not be released in time when operators use managed memory frequently
[
https://issues.apache.org/jira/browse/FLINK-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276114#comment-17276114
]
Caizhi Weng commented on FLINK-20663:
-
Hi,
I've tested the patch [~xintongsong] provided on some large batch SQL jobs and
the memory problem didn't occur anymore (these jobs will frequently fail with
memory problem every 10 minutes or so without the patch), so from the test
results I think this is a decent solution.
> Managed memory may not be released in time when operators use managed memory
> frequently
> ---
>
> Key: FLINK-20663
> URL: https://issues.apache.org/jira/browse/FLINK-20663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.12.2
>
>
> Some batch operators (like sort merge join or hash aggregate) use managed
> memory frequently. When these operators are chained together and the cluster
> load is a bit heavy, it is very likely that the following exception occurs:
> {code:java}
> 2020-12-18 10:04:32
> java.lang.RuntimeException:
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate
> 512 pages
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:85)
> at
> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.(SimpleCollectingOutputView.java:49)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.(BytesHashMap.java:297)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:103)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:90)
> at LocalHashAggregateWithKeys$209161.open(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:506)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:834)
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209161.close(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.TableStreamOperator.dispose(TableStreamOperator.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:739)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:719)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:642)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:551)
> ... 3 more
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209766.close(Unknown
> Source)
> ... 8 more
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 512 pages
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:231)
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:83)
> ... 13 more
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 16777216 bytes, only 9961487 bytes are remaining. This usually
> indicates that you are requesting more memory than you have reserved.
> However, when running an old JVM version it can also be caused by slow
> garbage collection. Try to upgrade to Java 8u72 or higher if running on an
> old Java version.
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:164)
> at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:80)
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:229)
> ... 14 more
> {code}
> It seems that this is caused by relying on GC to release managed memory, as
> {{System.gc()}} may not trigger GC in time. See {{UnsafeMemoryBudget.java}}.
--
This message was sent by
[jira] [Assigned] (FLINK-21150) Introduce ExecEdge to connect two ExecNodes
[ https://issues.apache.org/jira/browse/FLINK-21150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he reassigned FLINK-21150: -- Assignee: godfrey he > Introduce ExecEdge to connect two ExecNodes > --- > > Key: FLINK-21150 > URL: https://issues.apache.org/jira/browse/FLINK-21150 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #14822: [FLINK-21005][table-api] Introduce new provider for unified Sink API and implement in planner
wuchong commented on a change in pull request #14822:
URL: https://github.com/apache/flink/pull/14822#discussion_r567614908
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecSink.java
##
@@ -123,6 +126,10 @@ public CommonExecSink(
final DataStreamSinkProvider provider =
(DataStreamSinkProvider) runtimeProvider;
return
provider.consumeDataStream(dataStream).getTransformation();
}
+} else if (runtimeProvider instanceof SinkProvider) {
+final Sink sink = ((SinkProvider)
runtimeProvider).createSink();
+return new SinkTransformation<>(
Review comment:
We should also add not-null enfore logic `SinkNotNullEnforcer`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-21224) Remove BatchExecExchange and StreamExecExchange, and replace their functionality with ExecEdge
godfrey he created FLINK-21224: -- Summary: Remove BatchExecExchange and StreamExecExchange, and replace their functionality with ExecEdge Key: FLINK-21224 URL: https://issues.apache.org/jira/browse/FLINK-21224 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: godfrey he Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #14822: [FLINK-21005][table-api] Introduce new provider for unified Sink API and implement in planner
wuchong commented on a change in pull request #14822:
URL: https://github.com/apache/flink/pull/14822#discussion_r567614908
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecSink.java
##
@@ -123,6 +126,10 @@ public CommonExecSink(
final DataStreamSinkProvider provider =
(DataStreamSinkProvider) runtimeProvider;
return
provider.consumeDataStream(dataStream).getTransformation();
}
+} else if (runtimeProvider instanceof SinkProvider) {
+final Sink sink = ((SinkProvider)
runtimeProvider).createSink();
+return new SinkTransformation<>(
Review comment:
We should also add enfore logic `SinkNotNullEnforcer`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #14822: [FLINK-21005][table-api] Introduce new provider for unified Sink API and implement in planner
wuchong commented on a change in pull request #14822:
URL: https://github.com/apache/flink/pull/14822#discussion_r567610927
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/sink/SinkProvider.java
##
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.connector.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.table.data.RowData;
+
+/** Provider of a {@link Sink} instance as a runtime implementation for {@link
DynamicTableSink}. */
+@PublicEvolving
+public interface SinkProvider extends DynamicTableSink.SinkRuntimeProvider {
Review comment:
extends `ParallelismProvider` as well?
If we supports `ParallelismProvider` in this version, we should also
consider the keyBy shuffle when paralleslim is not the same to the upstream
operator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #14822: [FLINK-21005][table-api] Introduce new provider for unified Sink API and implement in planner
wuchong commented on a change in pull request #14822:
URL: https://github.com/apache/flink/pull/14822#discussion_r567610927
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/sink/SinkProvider.java
##
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.connector.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.table.data.RowData;
+
+/** Provider of a {@link Sink} instance as a runtime implementation for {@link
DynamicTableSink}. */
+@PublicEvolving
+public interface SinkProvider extends DynamicTableSink.SinkRuntimeProvider {
Review comment:
extends `ParallelismProvider` as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dianfu merged pull request #14825: [FLINK-21216][python] Limit numpy version in setup.py
dianfu merged pull request #14825: URL: https://github.com/apache/flink/pull/14825 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14826: [FLINK-12358][docs] Add IT case to verify REST docs corresponding to the code
flinkbot edited a comment on pull request #14826: URL: https://github.com/apache/flink/pull/14826#issuecomment-770627972 ## CI report: * ce1c42b9e4162b61553f3fadd02d30107047f91d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12724) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
flinkbot edited a comment on pull request #14647: URL: https://github.com/apache/flink/pull/14647#issuecomment-760154591 ## CI report: * 80e3121418e741d9bebf3e849817a0c4225a86dd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12587) * 8f5f9f44b437922f5e0edd0b1dd3097a61351dee Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12723) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21221) Deduplication of ResourceCounter
[
https://issues.apache.org/jira/browse/FLINK-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-21221:
-
Labels: starter (was: )
> Deduplication of ResourceCounter
>
>
> Key: FLINK-21221
> URL: https://issues.apache.org/jira/browse/FLINK-21221
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yangze Guo
>Priority: Major
> Labels: starter
>
> Currently, we have two {{ResourceCounter}} in our code base:
> - org.apache.flink.runtime.jobmaster.slotpool.ResourceCounter. Used in
> SlotPool component.
> - org.apache.flink.runtime.slots.ResourceCounter. Used in SlotManager
> component.
> Functionally speaking, these two {{ResourceCounter}} are the same. We can
> merge them to one class for deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21220) Change the type of cpuCores in ResourceProfile to CPUResource
[
https://issues.apache.org/jira/browse/FLINK-21220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-21220:
-
Labels: starter (was: )
> Change the type of cpuCores in ResourceProfile to CPUResource
> -
>
> Key: FLINK-21220
> URL: https://issues.apache.org/jira/browse/FLINK-21220
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yangze Guo
>Priority: Minor
> Labels: starter
>
> The {{cpuCores}} field in {{ResourceProfile}} should always be the
> CPUResource. There is no need to generalize it to {{Resource}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21223) Support to specify the input/output types of Python UDFs via string
Dian Fu created FLINK-21223:
---
Summary: Support to specify the input/output types of Python UDFs
via string
Key: FLINK-21223
URL: https://issues.apache.org/jira/browse/FLINK-21223
Project: Flink
Issue Type: Improvement
Components: API / Python
Reporter: Dian Fu
Currently, users need to specify the input/output types as following:
{code}
{{@udf(result_type=DataTypes.BIGINT())
def add(i, j):
return i + j
}}{code}
[FLIP-65|https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs]
makes it possible to support syntaxes as following:
{code}
{{@udf(result_type="BIGINT")
def add(i, j):
return i + j
}}{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16224) Refine Hadoop Delegation Token based testing framework
[ https://issues.apache.org/jira/browse/FLINK-16224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276104#comment-17276104 ] Matthias commented on FLINK-16224: -- Hi [~jackwangcs], I don't know of any plans to work on this soon. > Refine Hadoop Delegation Token based testing framework > -- > > Key: FLINK-16224 > URL: https://issues.apache.org/jira/browse/FLINK-16224 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Rong Rong >Priority: Major > > Currently the SecureTestEnvironment doesn't support Hadoop delegation token, > which makes the E2E testing of delegation-token-based YARN application > impossible. > Propose to enhance the testing framework to support delegation token based > launch in YARN cluster -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14826: [FLINK-12358][docs] Add IT case to verify REST docs corresponding to the code
flinkbot commented on pull request #14826: URL: https://github.com/apache/flink/pull/14826#issuecomment-770627972 ## CI report: * ce1c42b9e4162b61553f3fadd02d30107047f91d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * e97b4de046ec8231040f2c88b851a3fd69bf647b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12720) * b5045c1695a3b0260a34d85077687b50a198a9cd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12722) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14678: [FLINK-20833][runtime] Add pluggable failure listener in job manager
flinkbot edited a comment on pull request #14678: URL: https://github.com/apache/flink/pull/14678#issuecomment-761905721 ## CI report: * cade20e85b29ca63c51383dca04976c1d9801042 UNKNOWN * 12652d9c7829d2d26d41f61185cbd7ab8f4ad505 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12719) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
flinkbot edited a comment on pull request #14647: URL: https://github.com/apache/flink/pull/14647#issuecomment-760154591 ## CI report: * 80e3121418e741d9bebf3e849817a0c4225a86dd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12587) * 8f5f9f44b437922f5e0edd0b1dd3097a61351dee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21222) Support loopback mode to allow Python UDF worker and client reuse the same Python VM
Dian Fu created FLINK-21222: --- Summary: Support loopback mode to allow Python UDF worker and client reuse the same Python VM Key: FLINK-21222 URL: https://issues.apache.org/jira/browse/FLINK-21222 Project: Flink Issue Type: Improvement Components: API / Python Reporter: Dian Fu Currently the Python process for the client and the Python UDF worker runs in different Python processes. It would bring the following benefits if we could support loopback mode to allow Python UDF worker and client reuse the same Python VM: 1) It could greatly reduce the job initialization overhead in mini-cluster mode as there is no need to start the Python process any more. 2) It makes the debug easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14826: [FLINK-12358][docs] Add IT case to verify REST docs corresponding to the code
flinkbot commented on pull request #14826: URL: https://github.com/apache/flink/pull/14826#issuecomment-770608140 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ce1c42b9e4162b61553f3fadd02d30107047f91d (Mon Feb 01 06:33:12 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-21215) UnalignedCheckpointITCase.execute Failed
[ https://issues.apache.org/jira/browse/FLINK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reassigned FLINK-21215: -- Assignee: Piotr Nowojski > UnalignedCheckpointITCase.execute Failed > > > Key: FLINK-21215 > URL: https://issues.apache.org/jira/browse/FLINK-21215 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.13.0 >Reporter: Guowei Ma >Assignee: Piotr Nowojski >Priority: Major > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12691=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9146] > > > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:98) > > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:84) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1930) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1007) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$9(SchedulerBase.java:1009) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$processCheckpointCoordinatorMessage$10(SchedulerBase.java:1025) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[ https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276088#comment-17276088 ] Jark Wu commented on FLINK-21045: - Thanks [~qingyue] for the nice summary. Let's continue the discussion in ML. > Support 'load module' and 'unload module' SQL syntax > > > Key: FLINK-21045 > URL: https://issues.apache.org/jira/browse/FLINK-21045 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.13.0 >Reporter: Nicholas Jiang >Assignee: Jane Chan >Priority: Major > Fix For: 1.13.0 > > > At present, Flink SQL doesn't support the 'load module' and 'unload module' > SQL syntax. It's necessary for uses in the situation that users load and > unload user-defined module through table api or sql client. > SQL syntax has been proposed in FLIP-68: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12358) Verify whether rest documenation needs to be updated when building pull request
[ https://issues.apache.org/jira/browse/FLINK-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-12358: --- Labels: pull-request-available (was: ) > Verify whether rest documenation needs to be updated when building pull > request > --- > > Key: FLINK-12358 > URL: https://issues.apache.org/jira/browse/FLINK-12358 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > > Currently, unlike configuration docs, rest-API docs have no any methods to > check whether updated to latest code. This is really annoying and not easy to > track if only checked by developers. > I plan to check this in travis to verify whether any files have been updated > by using `git status`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #14813: [FLINK-21207] Fix source table with 'csv.disable-quote-character' = 'true' can not take effect
wuchong commented on a change in pull request #14813:
URL: https://github.com/apache/flink/pull/14813#discussion_r567589468
##
File path:
flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatFactoryTest.java
##
@@ -206,6 +206,29 @@ public void testDeserializeWithEscapedFieldDelimiter()
throws IOException {
assertEquals(expected, actual);
}
+@Test
+public void testDeserializeWithDisableQuoteCharacter() throws IOException {
Review comment:
`CsvFormatFactoryTest` is used to verify the received options can be
converted into expected serializer/deserializer. The parse/unparse logic should
be tested in `CsvRowDataSerDeSchemaTest`.
Could you add a test in `CsvRowDataSerDeSchemaTest` to test deserialization
with quote disabled ? And add a DeserializationSchema assertEquals test in
`CsvFormatFactoryTest#testDisableQuoteCharacter`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Myasuka opened a new pull request #14826: [FLINK-12358][docs] Add IT case to verify REST docs corresponding to the code
Myasuka opened a new pull request #14826: URL: https://github.com/apache/flink/pull/14826 ## What is the purpose of the change Include method/path/query parameter description in the `rest_api_v1.snapshot` so that we could follow the description check. Add IT case `RestOptionsDocsCompletenessITCase` to ensure the `rest_v1_dispatcher.html` corresponds to the `rest_api_v1.snapshot`. ## Brief change log - Regenerate REST API html doc. - Include method/path/query parameter description in the `rest_api_v1.snapshot` so that we could follow the description check. - Add IT case `RestOptionsDocsCompletenessITCase` to ensure the `rest_v1_dispatcher.html` corresponds to the `rest_api_v1.snapshot`. ## Verifying this change This change added tests and can be verified as follows: - Newly added integration tests `RestOptionsDocsCompletenessITCase` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-14256) [Umbrella] Introduce FileSystemTableFactory with partitioned support
[ https://issues.apache.org/jira/browse/FLINK-14256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276084#comment-17276084 ] Zili Chen commented on FLINK-14256: --- Plz update FLIP status correspondingly [~lzljs3620320]. > [Umbrella] Introduce FileSystemTableFactory with partitioned support > > > Key: FLINK-14256 > URL: https://issues.apache.org/jira/browse/FLINK-14256 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.11.0 > > > Introduce FileSystemTableFactory to unify all file system connectors. > More information in > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20722) HiveTableSink should copy the record when converting RowData to Row
[
https://issues.apache.org/jira/browse/FLINK-20722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276080#comment-17276080
]
Jark Wu commented on FLINK-20722:
-
Sorry, but shouldn't this be fixed in
{{multipleinput.output.BroadcastingOutput}}? Otherwise, all the connectors may
need this special fix which is introduced by multi-input. IMO, multi-input
should create a copying {{BroadcastingOutput}} when object reuse is enabled,
just like how runtime operator chian does,
{{CopyingBroadcastingOutputCollector}} vs. {{BroadcastingOutputCollector}}.
What do you think [~godfreyhe]?
> HiveTableSink should copy the record when converting RowData to Row
> ---
>
> Key: FLINK-20722
> URL: https://issues.apache.org/jira/browse/FLINK-20722
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Reporter: Rui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Add the following test in {{TableEnvHiveConnectorITCase}} to reproduce the
> issue:
> {code}
> @Test
> public void test() throws Exception {
> TableEnvironment tableEnv = getTableEnvWithHiveCatalog();
> tableEnv.executeSql("create table src1(key string, val
> string)");
> tableEnv.executeSql("create table src2(key string, val
> string)");
> tableEnv.executeSql("create table dest(key string, val
> string)");
> HiveTestUtils.createTextTableInserter(hiveCatalog, "default",
> "src1")
> .addRow(new Object[]{"1", "val1"})
> .addRow(new Object[]{"2", "val2"})
> .addRow(new Object[]{"3", "val3"})
> .commit();
> HiveTestUtils.createTextTableInserter(hiveCatalog, "default",
> "src2")
> .addRow(new Object[]{"3", "val4"})
> .addRow(new Object[]{"4", "val4"})
> .commit();
> tableEnv.executeSql("INSERT OVERWRITE dest\n" +
> "SELECT j.*\n" +
> "FROM (SELECT t1.key, p1.val\n" +
> " FROM src2 t1\n" +
> " LEFT OUTER JOIN src1 p1\n" +
> " ON (t1.key = p1.key)\n" +
> " UNION ALL\n" +
> " SELECT t2.key, p2.val\n" +
> " FROM src2 t2\n" +
> " LEFT OUTER JOIN src1 p2\n" +
> " ON (t2.key = p2.key)) j").await();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * e97b4de046ec8231040f2c88b851a3fd69bf647b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12720) * b5045c1695a3b0260a34d85077687b50a198a9cd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21203) Don’t collect -U&+U Row When they are equals In the LastRowFunction
[
https://issues.apache.org/jira/browse/FLINK-21203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276078#comment-17276078
]
wangpeibin commented on FLINK-21203:
+1 for Jark's suggestion
> Don’t collect -U&+U Row When they are equals In the LastRowFunction
> -
>
> Key: FLINK-21203
> URL: https://issues.apache.org/jira/browse/FLINK-21203
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: wangpeibin
>Assignee: wangpeibin
>Priority: Major
>
> In the LastRowFunction , the -U&+U Row will be collected even if they are
> the same, which will increase calculation pressure of the next Operator.
>
> To avoid this, we can optimize the logic of DeduplicateFunctionHelper. Also,
> a config to enable the optimization will be added.
> With the sql followed:
> {quote}select * from
> (select
> *,
> row_number() over (partition by k order by proctime() desc ) as row_num
> from a
> ) t
> where row_num = 1
> {quote}
> Then input 2 row such as :
> {quote}Event("B","1","b"),
> Event("B","1","b")
> {quote}
> Now the output is:
> {quote}(true,+I[B, 1, b, 1])
> (false,-U[B, 1, b, 1])
> (true,+U[B, 1, b, 1])
> {quote}
> After the optimization, the output will be:
> {quote}(true,+I[B, 1, b, 1])
> {quote}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20216) Move "Configuration" and "Environment Variables" under "Python API" section
[ https://issues.apache.org/jira/browse/FLINK-20216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-20216. --- Fix Version/s: (was: 1.13.0) 1.12.0 Resolution: Fixed > Move "Configuration" and "Environment Variables" under "Python API" section > --- > > Key: FLINK-20216 > URL: https://issues.apache.org/jira/browse/FLINK-20216 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Fix For: 1.12.0 > > > "Configuration" and "Environment Variables" are currently under the > documentation of "Python Table API". However, these sections apply for both > Python DataStream API and Python Table API and so we should move them under > "Python API". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #14813: [FLINK-21207] Fix source table with 'csv.disable-quote-character' = 'true' can not take effect
wuchong commented on a change in pull request #14813:
URL: https://github.com/apache/flink/pull/14813#discussion_r567584364
##
File path:
flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatFactoryTest.java
##
@@ -206,6 +206,29 @@ public void testDeserializeWithEscapedFieldDelimiter()
throws IOException {
assertEquals(expected, actual);
}
+@Test
+public void testDeserializeWithDisableQuoteCharacter() throws IOException {
+// test deserialization schema
+final Map options =
+getModifiedOptions(
+opts -> {
+opts.put("csv.disable-quote-character", "true");
+opts.remove("csv.quote-character");
+});
+
+final DynamicTableSource actualSource = createTableSource(options);
+assert actualSource instanceof
TestDynamicTableFactory.DynamicTableSourceMock;
+TestDynamicTableFactory.DynamicTableSourceMock sourceMock =
+(TestDynamicTableFactory.DynamicTableSourceMock) actualSource;
+
+DeserializationSchema deserializationSchema =
+sourceMock.valueFormat.createRuntimeDecoder(
+ScanRuntimeProviderContext.INSTANCE,
SCHEMA.toRowDataType());
+RowData expected = GenericRowData.of(fromString("\"abc"), 123, false);
+RowData actual =
deserializationSchema.deserialize("\"abc;123;false".getBytes());
Review comment:
Thanks for the explanation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
KarmaGYZ commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r567581423
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/FineGrainedSlotManager.java
##
@@ -0,0 +1,790 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager.slotmanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotID;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.concurrent.ScheduledExecutor;
+import org.apache.flink.runtime.instance.InstanceID;
+import org.apache.flink.runtime.metrics.MetricNames;
+import org.apache.flink.runtime.metrics.groups.SlotManagerMetricGroup;
+import org.apache.flink.runtime.resourcemanager.ResourceManagerId;
+import org.apache.flink.runtime.resourcemanager.WorkerResourceSpec;
+import
org.apache.flink.runtime.resourcemanager.registration.TaskExecutorConnection;
+import org.apache.flink.runtime.slots.ResourceCounter;
+import org.apache.flink.runtime.slots.ResourceRequirement;
+import org.apache.flink.runtime.slots.ResourceRequirements;
+import org.apache.flink.runtime.taskexecutor.SlotReport;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/** Implementation of {@link SlotManager} supporting fine-grained resource
management. */
+public class FineGrainedSlotManager implements SlotManager {
+private static final Logger LOG =
LoggerFactory.getLogger(FineGrainedSlotManager.class);
+
+private final TaskManagerTracker taskManagerTracker;
+private final ResourceTracker resourceTracker;
+private final ResourceAllocationStrategy resourceAllocationStrategy;
+
+private final SlotStatusSyncer slotStatusSyncer;
+
+/** Scheduled executor for timeouts. */
+private final ScheduledExecutor scheduledExecutor;
+
+/** Timeout after which an unused TaskManager is released. */
+private final Time taskManagerTimeout;
+
+private final SlotManagerMetricGroup slotManagerMetricGroup;
+
+private final Map jobMasterTargetAddresses = new
HashMap<>();
+
+/** Defines the max limitation of the total number of task executors. */
+private final int maxTaskManagerNum;
+
+/** Defines the number of redundant task executors. */
+private final int redundantTaskManagerNum;
+
+/**
+ * Release task executor only when each produced result partition is
either consumed or failed.
+ */
+private final boolean waitResultConsumedBeforeRelease;
+
+/** The default resource spec of workers to request. */
+private final WorkerResourceSpec defaultWorkerResourceSpec;
+
+/** The resource profile of default slot. */
+private final ResourceProfile defaultSlotResourceProfile;
+
+private boolean sendNotEnoughResourceNotifications = true;
+
+/** ResourceManager's id. */
+@Nullable private ResourceManagerId resourceManagerId;
+
+/** Executor for future callbacks which have to be "synchronized". */
+@Nullable private Executor mainThreadExecutor;
+
+/** Callbacks for resource (de-)allocations. */
+@Nullable private ResourceActions resourceActions;
+
+private ScheduledFuture
[GitHub] [flink] flinkbot edited a comment on pull request #14825: [FLINK-21216][python] Limit numpy version in setup.py
flinkbot edited a comment on pull request #14825: URL: https://github.com/apache/flink/pull/14825#issuecomment-770581870 ## CI report: * 291f6797c35037d17e9aeb8ab67da557d5c35b87 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12721) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14824: [release-1.12][FLINK-21169][kafka] flink-connector-base dependency should be scope compile
flinkbot edited a comment on pull request #14824: URL: https://github.com/apache/flink/pull/14824#issuecomment-770512589 ## CI report: * e434ce611d8c12cceb749d22ea24f3e6fb650aeb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12718) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20495) Elasticsearch6DynamicSinkITCase Hang
[
https://issues.apache.org/jira/browse/FLINK-20495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276069#comment-17276069
]
Xintong Song commented on FLINK-20495:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12720=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20
> Elasticsearch6DynamicSinkITCase Hang
>
>
> Key: FLINK-20495
> URL: https://issues.apache.org/jira/browse/FLINK-20495
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Connectors /
> ElasticSearch, Tests
>Affects Versions: 1.13.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10535=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20]
>
> {code:java}
> 2020-12-04T22:39:33.9748225Z [INFO] Running
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase
> 2020-12-04T22:54:51.9486410Z
> ==
> 2020-12-04T22:54:51.9488766Z Process produced no output for 900 seconds.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-21171) Introduce TypedValue to the StateFun request-reply protocol
[
https://issues.apache.org/jira/browse/FLINK-21171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai closed FLINK-21171.
---
Resolution: Fixed
flink-statefun/master: 69c658ec361682fb3bef50bd810f0646d7332a0c
> Introduce TypedValue to the StateFun request-reply protocol
> ---
>
> Key: FLINK-21171
> URL: https://issues.apache.org/jira/browse/FLINK-21171
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Labels: pull-request-available
> Fix For: statefun-3.0.0
>
>
> Currently, all values being passed around via the request-reply protocol, are
> of the Protobuf {{Any}} type. This includes payloads of outgoing messages to
> other functions, and also state values.
> This has a few shortcomings:
> * All user records are strictly required to be modeled and wrapped as a
> Protobuf message - even for simple primitive types. This makes it awkward to
> work with for many common types of messages, for example JSON.
> * For data persisted as state, with each state value being a Protobuf
> {{Any}}, each value would also redundantly store the type urls associated
> with each Protobuf message.
> Instead, we'd like to introduce a {{TypedValue}} construct that replaces
> {{Any}} everywhere in the protocol, for both messages and state values:
> {code}
> message TypedValue {
> string typename = 1;
> bytes value = 2;
> }
> {code}
> The {{typename}} here directly maps to the type concept introduced in
> FLINK-21061.
> For state, we directly write the value bytes of a {{TypedValue}} into state,
> and the {{typename}} is the meta information snapshotted by the state
> serializer (see FLINK-21061).
> For messages, the new {{TypedValue}} opens up the possibility for
> user-defined types. For example, a user can serialize a JSON string as the
> value bytes, and define a custom typename to indentify the type.
> We can also leverage this to define built-in types, for example a
> cross-language unification of primitive types. This would be an extended
> scope of this ticket and will be separately handled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai closed pull request #195: [FLINK-21171] Wire in TypedValue throughout the runtime as state values and message payloads
tzulitai closed pull request #195: URL: https://github.com/apache/flink-statefun/pull/195 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21203) Don’t collect -U&+U Row When they are equals In the LastRowFunction
[
https://issues.apache.org/jira/browse/FLINK-21203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276066#comment-17276066
]
Jark Wu commented on FLINK-21203:
-
I think we may not need the configuration. Group Aggregate also has the similar
logic when state ttl is not enabled.
We may need to enable this optimization only when state ttl is disabled, just
like the implementation in
https://github.com/apache/flink/blob/f3db4220f5c8730e065734cff16237c7743b390f/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/aggregate/GroupAggFunction.java#L170
> Don’t collect -U&+U Row When they are equals In the LastRowFunction
> -
>
> Key: FLINK-21203
> URL: https://issues.apache.org/jira/browse/FLINK-21203
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: wangpeibin
>Assignee: wangpeibin
>Priority: Major
>
> In the LastRowFunction , the -U&+U Row will be collected even if they are
> the same, which will increase calculation pressure of the next Operator.
>
> To avoid this, we can optimize the logic of DeduplicateFunctionHelper. Also,
> a config to enable the optimization will be added.
> With the sql followed:
> {quote}select * from
> (select
> *,
> row_number() over (partition by k order by proctime() desc ) as row_num
> from a
> ) t
> where row_num = 1
> {quote}
> Then input 2 row such as :
> {quote}Event("B","1","b"),
> Event("B","1","b")
> {quote}
> Now the output is:
> {quote}(true,+I[B, 1, b, 1])
> (false,-U[B, 1, b, 1])
> (true,+U[B, 1, b, 1])
> {quote}
> After the optimization, the output will be:
> {quote}(true,+I[B, 1, b, 1])
> {quote}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14825: [FLINK-21216][python] Fix numpy version in setup.py
flinkbot commented on pull request #14825: URL: https://github.com/apache/flink/pull/14825#issuecomment-770581870 ## CI report: * 291f6797c35037d17e9aeb8ab67da557d5c35b87 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14825: [FLINK-21216][python] Fix numpy version in setup.py
flinkbot commented on pull request #14825: URL: https://github.com/apache/flink/pull/14825#issuecomment-770576435 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 291f6797c35037d17e9aeb8ab67da557d5c35b87 (Mon Feb 01 05:26:55 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21216).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21216) StreamPandasConversionTests Fails
[ https://issues.apache.org/jira/browse/FLINK-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21216: --- Labels: pull-request-available test-stability (was: test-stability) > StreamPandasConversionTests Fails > - > > Key: FLINK-21216 > URL: https://issues.apache.org/jira/browse/FLINK-21216 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.12.0, 1.13.0 >Reporter: Guowei Ma >Priority: Major > Labels: pull-request-available, test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12699=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3] > > === FAILURES > === > ___ StreamPandasConversionTests.test_empty_to_pandas > ___ > > self = > testMethod=test_empty_to_pandas> > > def test_empty_to_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type) > > pyflink/table/tests/test_pandas_conversion.py:144: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > pyflink/table/table_environment.py:1462: in from_pandas > arrow_schema = pa.Schema.from_pandas(pdf, preserve_index=False) > pyarrow/types.pxi:1315: in pyarrow.lib.Schema.from_pandas > ??? > .tox/py37-cython/lib/python3.7/site-packages/pyarrow/pandas_compat.py:519: in > dataframe_to_types > type_ = pa.lib._ndarray_to_arrow_type(values, type_) > pyarrow/array.pxi:53: in pyarrow.lib._ndarray_to_arrow_type > ??? > pyarrow/array.pxi:64: in pyarrow.lib._ndarray_to_type > ??? > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > > > ??? > E pyarrow.lib.ArrowTypeError: Did not pass numpy.dtype object > > pyarrow/error.pxi:108: ArrowTypeError > _ StreamPandasConversionTests.test_from_pandas > _ > > self = > testMethod=test_from_pandas> > > def test_from_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type, 5) > > pyflink/table/tests/test_pandas_conversion.py:120: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] HuangXingBo opened a new pull request #14825: [FLINK-21216][python] Fix numpy version in setup.py
HuangXingBo opened a new pull request #14825: URL: https://github.com/apache/flink/pull/14825 ## What is the purpose of the change *This pull request fix numpy version in setup.py. Currently pyarrow>=0.15.1,<0.18.0 and numpy==1.20.0 have compatibility problem in Python 3.7.* ## Brief change log - *set numpy version >=1.14.3,<1.20* ## Verifying this change This change added tests and can be verified as follows: - *original tests* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * dfd5c341320f3ccb7291c5833ae52a3d3f1632f1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12716) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
curcur commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r567563070
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
##
@@ -44,6 +44,15 @@
.noDefaultValue()
.withDescription("The state backend to be used to
checkpoint state.");
+/** Whether to enable state change log. */
+// @Documentation.Section(value =
Documentation.Sections.COMMON_STATE_BACKENDS)
+public static final ConfigOption ENABLE_STATE_CHANGE_LOG =
+ConfigOptions.key("state.backend.enable-statechangelog")
Review comment:
do you mean we should hide it from users for now or we should not use
this option?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
curcur commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r567556094
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/proxy/ProxyKeyedStateBackend.java
##
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.proxy;
+
+import org.apache.flink.api.common.state.AggregatingStateDescriptor;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ReducingStateDescriptor;
+import org.apache.flink.api.common.state.State;
+import org.apache.flink.api.common.state.StateDescriptor;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue;
+import org.apache.flink.runtime.state.Keyed;
+import org.apache.flink.runtime.state.KeyedStateFunction;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.PriorityComparable;
+import org.apache.flink.runtime.state.SnapshotResult;
+import org.apache.flink.runtime.state.StateSnapshotTransformer;
+import org.apache.flink.runtime.state.heap.HeapKeyedStateBackend;
+import org.apache.flink.runtime.state.heap.HeapPriorityQueueElement;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nonnull;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RunnableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/** */
+public class ProxyKeyedStateBackend extends AbstractKeyedStateBackend {
+// wrapped keyed state backend, either HeapKeyedStateBackend or
RocksDBKeyedStateBackend
+AbstractKeyedStateBackend keyedStateBackend;
+
+private static final Map, StateFactory>
STATE_FACTORIES =
+Stream.of(
+Tuple2.of(
+ValueStateDescriptor.class,
+(StateFactory) ProxyValueState::create),
+Tuple2.of(
+ListStateDescriptor.class,
+(StateFactory) ProxyListState::create),
+Tuple2.of(
+ReducingStateDescriptor.class,
+(StateFactory) ProxyReducingState::create),
+Tuple2.of(
+AggregatingStateDescriptor.class,
+(StateFactory)
ProxyAggregatingState::create),
+Tuple2.of(
+MapStateDescriptor.class, (StateFactory)
ProxyMapState::create))
+.collect(Collectors.toMap(t -> t.f0, t -> t.f1));
+
+public ProxyKeyedStateBackend(AbstractKeyedStateBackend
keyedStateBackend) {
+super(
+keyedStateBackend.kvStateRegistry,
+keyedStateBackend.keySerializer,
+keyedStateBackend.userCodeClassLoader,
+keyedStateBackend.executionConfig,
+keyedStateBackend.ttlTimeProvider,
+keyedStateBackend.cancelStreamRegistry,
+keyedStateBackend.keyGroupCompressionDecorator,
+keyedStateBackend.keyContext);
+this.keyedStateBackend = keyedStateBackend;
Review comment:
All these questions are great! For all the reasons mentioned or not
mentioned above, I feel this hierarchy is confusing as well. So I've spent some
time during the weekend simplifying it.
Originally, the reason I extend `AbstractKeyedStateBackend` is I thought all
fields have to be wrapped and delegated.
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * e97b4de046ec8231040f2c88b851a3fd69bf647b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12720) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sharkdtu commented on a change in pull request #14813: [FLINK-21207] Fix source table with 'csv.disable-quote-character' = 'true' can not take effect
sharkdtu commented on a change in pull request #14813:
URL: https://github.com/apache/flink/pull/14813#discussion_r567551672
##
File path:
flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatFactoryTest.java
##
@@ -206,6 +206,29 @@ public void testDeserializeWithEscapedFieldDelimiter()
throws IOException {
assertEquals(expected, actual);
}
+@Test
+public void testDeserializeWithDisableQuoteCharacter() throws IOException {
+// test deserialization schema
+final Map options =
+getModifiedOptions(
+opts -> {
+opts.put("csv.disable-quote-character", "true");
+opts.remove("csv.quote-character");
+});
+
+final DynamicTableSource actualSource = createTableSource(options);
+assert actualSource instanceof
TestDynamicTableFactory.DynamicTableSourceMock;
+TestDynamicTableFactory.DynamicTableSourceMock sourceMock =
+(TestDynamicTableFactory.DynamicTableSourceMock) actualSource;
+
+DeserializationSchema deserializationSchema =
+sourceMock.valueFormat.createRuntimeDecoder(
+ScanRuntimeProviderContext.INSTANCE,
SCHEMA.toRowDataType());
+RowData expected = GenericRowData.of(fromString("\"abc"), 123, false);
+RowData actual =
deserializationSchema.deserialize("\"abc;123;false".getBytes());
Review comment:
It can throw exception when 'csv.disable-quote-character' is set false
(default value), because it does not find closed quota of `"abc`. But when
'csv.disable-quote-character' is set ture, it shouldn't throw excetpion as it
will ignore quote character. Actually, this didn't take effect. So this pr fix
this bug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] sharkdtu commented on a change in pull request #14813: [FLINK-21207] Fix source table with 'csv.disable-quote-character' = 'true' can not take effect
sharkdtu commented on a change in pull request #14813:
URL: https://github.com/apache/flink/pull/14813#discussion_r567551672
##
File path:
flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatFactoryTest.java
##
@@ -206,6 +206,29 @@ public void testDeserializeWithEscapedFieldDelimiter()
throws IOException {
assertEquals(expected, actual);
}
+@Test
+public void testDeserializeWithDisableQuoteCharacter() throws IOException {
+// test deserialization schema
+final Map options =
+getModifiedOptions(
+opts -> {
+opts.put("csv.disable-quote-character", "true");
+opts.remove("csv.quote-character");
+});
+
+final DynamicTableSource actualSource = createTableSource(options);
+assert actualSource instanceof
TestDynamicTableFactory.DynamicTableSourceMock;
+TestDynamicTableFactory.DynamicTableSourceMock sourceMock =
+(TestDynamicTableFactory.DynamicTableSourceMock) actualSource;
+
+DeserializationSchema deserializationSchema =
+sourceMock.valueFormat.createRuntimeDecoder(
+ScanRuntimeProviderContext.INSTANCE,
SCHEMA.toRowDataType());
+RowData expected = GenericRowData.of(fromString("\"abc"), 123, false);
+RowData actual =
deserializationSchema.deserialize("\"abc;123;false".getBytes());
Review comment:
It can throw exception when 'csv.disable-quote-character' is set false
(default value), because it does not find closed quota
of "abc. But when 'csv.disable-quote-character' is set ture, it shouldn't
throw excetpion as it will ignore quote character. Actually, this didn't take
effect. So this pr fix this bug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] sharkdtu commented on a change in pull request #14813: [FLINK-21207] Fix source table with 'csv.disable-quote-character' = 'true' can not take effect
sharkdtu commented on a change in pull request #14813:
URL: https://github.com/apache/flink/pull/14813#discussion_r567551672
##
File path:
flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatFactoryTest.java
##
@@ -206,6 +206,29 @@ public void testDeserializeWithEscapedFieldDelimiter()
throws IOException {
assertEquals(expected, actual);
}
+@Test
+public void testDeserializeWithDisableQuoteCharacter() throws IOException {
+// test deserialization schema
+final Map options =
+getModifiedOptions(
+opts -> {
+opts.put("csv.disable-quote-character", "true");
+opts.remove("csv.quote-character");
+});
+
+final DynamicTableSource actualSource = createTableSource(options);
+assert actualSource instanceof
TestDynamicTableFactory.DynamicTableSourceMock;
+TestDynamicTableFactory.DynamicTableSourceMock sourceMock =
+(TestDynamicTableFactory.DynamicTableSourceMock) actualSource;
+
+DeserializationSchema deserializationSchema =
+sourceMock.valueFormat.createRuntimeDecoder(
+ScanRuntimeProviderContext.INSTANCE,
SCHEMA.toRowDataType());
+RowData expected = GenericRowData.of(fromString("\"abc"), 123, false);
+RowData actual =
deserializationSchema.deserialize("\"abc;123;false".getBytes());
Review comment:
It can throw exception when 'csv.disable-quote-character' is set false
(default value), because it does not find closed quotation
of '\"abc'. But when 'csv.disable-quote-character' is set ture, it
shouldn't throw excetpion as it will ignore quote character. Actually, this
didn't take effect. So this pr fix this bug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
curcur commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r567551460
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/proxy/ProxyKeyedStateBackend.java
##
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.proxy;
+
+import org.apache.flink.api.common.state.AggregatingStateDescriptor;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ReducingStateDescriptor;
+import org.apache.flink.api.common.state.State;
+import org.apache.flink.api.common.state.StateDescriptor;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue;
+import org.apache.flink.runtime.state.Keyed;
+import org.apache.flink.runtime.state.KeyedStateFunction;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.PriorityComparable;
+import org.apache.flink.runtime.state.SnapshotResult;
+import org.apache.flink.runtime.state.StateSnapshotTransformer;
+import org.apache.flink.runtime.state.heap.HeapKeyedStateBackend;
+import org.apache.flink.runtime.state.heap.HeapPriorityQueueElement;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nonnull;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RunnableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/** */
+public class ProxyKeyedStateBackend extends AbstractKeyedStateBackend {
+// wrapped keyed state backend, either HeapKeyedStateBackend or
RocksDBKeyedStateBackend
+AbstractKeyedStateBackend keyedStateBackend;
+
+private static final Map, StateFactory>
STATE_FACTORIES =
+Stream.of(
+Tuple2.of(
+ValueStateDescriptor.class,
+(StateFactory) ProxyValueState::create),
+Tuple2.of(
+ListStateDescriptor.class,
Review comment:
I do not have a preference in general, simpler is better.
However, I am constructing the factory map this way to follow the same
style/functionality existing keyedStateBackends use, since this is a wrapper
for them.
If you prefer to make them simpler, we probably should change them
altogether. Having two different styles may lead to more confusion (as the TTL
wrapper looks to me), but I guess we can do it in a different PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] curcur commented on a change in pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
curcur commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r567551460
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/proxy/ProxyKeyedStateBackend.java
##
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.proxy;
+
+import org.apache.flink.api.common.state.AggregatingStateDescriptor;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ReducingStateDescriptor;
+import org.apache.flink.api.common.state.State;
+import org.apache.flink.api.common.state.StateDescriptor;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue;
+import org.apache.flink.runtime.state.Keyed;
+import org.apache.flink.runtime.state.KeyedStateFunction;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.PriorityComparable;
+import org.apache.flink.runtime.state.SnapshotResult;
+import org.apache.flink.runtime.state.StateSnapshotTransformer;
+import org.apache.flink.runtime.state.heap.HeapKeyedStateBackend;
+import org.apache.flink.runtime.state.heap.HeapPriorityQueueElement;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nonnull;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RunnableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/** */
+public class ProxyKeyedStateBackend extends AbstractKeyedStateBackend {
+// wrapped keyed state backend, either HeapKeyedStateBackend or
RocksDBKeyedStateBackend
+AbstractKeyedStateBackend keyedStateBackend;
+
+private static final Map, StateFactory>
STATE_FACTORIES =
+Stream.of(
+Tuple2.of(
+ValueStateDescriptor.class,
+(StateFactory) ProxyValueState::create),
+Tuple2.of(
+ListStateDescriptor.class,
Review comment:
I do not have a preference in general, simpler is better.
However, I am constructing the factory map this way to follow the same
style/functionality existing keyedStateBackends use, since this is a wrapper
for them.
If you prefer to make them simpler, we probably should change them
altogether. Having two different styles may lead to more confusion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] curcur commented on pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
curcur commented on pull request #14799: URL: https://github.com/apache/flink/pull/14799#issuecomment-770546162 Thank you so much for reviewing, @rkhachatryan ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21221) Deduplication of ResourceCounter
[
https://issues.apache.org/jira/browse/FLINK-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276044#comment-17276044
]
Yangze Guo commented on FLINK-21221:
cc: [~trohrmann] [~chesnay]
> Deduplication of ResourceCounter
>
>
> Key: FLINK-21221
> URL: https://issues.apache.org/jira/browse/FLINK-21221
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Yangze Guo
>Priority: Major
>
> Currently, we have two {{ResourceCounter}} in our code base:
> - org.apache.flink.runtime.jobmaster.slotpool.ResourceCounter. Used in
> SlotPool component.
> - org.apache.flink.runtime.slots.ResourceCounter. Used in SlotManager
> component.
> Functionally speaking, these two {{ResourceCounter}} are the same. We can
> merge them to one class for deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21221) Deduplication of ResourceCounter
Yangze Guo created FLINK-21221:
--
Summary: Deduplication of ResourceCounter
Key: FLINK-21221
URL: https://issues.apache.org/jira/browse/FLINK-21221
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Reporter: Yangze Guo
Currently, we have two {{ResourceCounter}} in our code base:
- org.apache.flink.runtime.jobmaster.slotpool.ResourceCounter. Used in SlotPool
component.
- org.apache.flink.runtime.slots.ResourceCounter. Used in SlotManager component.
Functionally speaking, these two {{ResourceCounter}} are the same. We can merge
them to one class for deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21207) 'csv.disable-quote-character' can not take effect during deserialization
[
https://issues.apache.org/jira/browse/FLINK-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-21207:
Component/s: Table SQL / Ecosystem
Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> 'csv.disable-quote-character' can not take effect during deserialization
> -
>
> Key: FLINK-21207
> URL: https://issues.apache.org/jira/browse/FLINK-21207
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: sharkd tu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.2, 1.13.0
>
>
> source table with 'csv.disable-quote-character' = 'true' can not take effect.
>
> {code}
> Caused by:
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
> Missing closing quote for valueCaused by:
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
> Missing closing quote for value at [Source: UNKNOWN; line: 1, column: 29] at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._reportParsingError(CsvParser.java:1250)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder._nextQuotedString(CsvDecoder.java:785)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder.nextString(CsvDecoder.java:630)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:846)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser.nextFieldName(CsvParser.java:665)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1282)
> at
> org.apache.flink.formats.csv.CsvRowDataDeserializationSchema.deserialize(CsvRowDataDeserializationSchema.java:155)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #14813: [FLINK-21207] Fix source table with 'csv.disable-quote-character' = 'true' can not take effect
wuchong commented on a change in pull request #14813:
URL: https://github.com/apache/flink/pull/14813#discussion_r567548368
##
File path:
flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatFactoryTest.java
##
@@ -206,6 +206,29 @@ public void testDeserializeWithEscapedFieldDelimiter()
throws IOException {
assertEquals(expected, actual);
}
+@Test
+public void testDeserializeWithDisableQuoteCharacter() throws IOException {
+// test deserialization schema
+final Map options =
+getModifiedOptions(
+opts -> {
+opts.put("csv.disable-quote-character", "true");
+opts.remove("csv.quote-character");
+});
+
+final DynamicTableSource actualSource = createTableSource(options);
+assert actualSource instanceof
TestDynamicTableFactory.DynamicTableSourceMock;
+TestDynamicTableFactory.DynamicTableSourceMock sourceMock =
+(TestDynamicTableFactory.DynamicTableSourceMock) actualSource;
+
+DeserializationSchema deserializationSchema =
+sourceMock.valueFormat.createRuntimeDecoder(
+ScanRuntimeProviderContext.INSTANCE,
SCHEMA.toRowDataType());
+RowData expected = GenericRowData.of(fromString("\"abc"), 123, false);
+RowData actual =
deserializationSchema.deserialize("\"abc;123;false".getBytes());
Review comment:
Shouldn't this throw exception because we quote is disabled
`csv.disable-quote-character=true`.
##
File path:
flink-formats/flink-csv/src/test/java/org/apache/flink/formats/csv/CsvFormatFactoryTest.java
##
@@ -206,6 +206,29 @@ public void testDeserializeWithEscapedFieldDelimiter()
throws IOException {
assertEquals(expected, actual);
}
+@Test
+public void testDeserializeWithDisableQuoteCharacter() throws IOException {
+// test deserialization schema
+final Map options =
+getModifiedOptions(
+opts -> {
+opts.put("csv.disable-quote-character", "true");
+opts.remove("csv.quote-character");
+});
+
+final DynamicTableSource actualSource = createTableSource(options);
+assert actualSource instanceof
TestDynamicTableFactory.DynamicTableSourceMock;
+TestDynamicTableFactory.DynamicTableSourceMock sourceMock =
+(TestDynamicTableFactory.DynamicTableSourceMock) actualSource;
+
+DeserializationSchema deserializationSchema =
+sourceMock.valueFormat.createRuntimeDecoder(
+ScanRuntimeProviderContext.INSTANCE,
SCHEMA.toRowDataType());
+RowData expected = GenericRowData.of(fromString("\"abc"), 123, false);
+RowData actual =
deserializationSchema.deserialize("\"abc;123;false".getBytes());
Review comment:
Shouldn't this throw exception because we quote is disabled
`csv.disable-quote-character=true`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-21216) StreamPandasConversionTests Fails
[ https://issues.apache.org/jira/browse/FLINK-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276042#comment-17276042 ] Guowei Ma edited comment on FLINK-21216 at 2/1/21, 3:37 AM: [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12713=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=f5211ead-5e53-5af8-f827-4dbf08df26bb] https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12713=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=613f538c-bcef-59e6-f9cd-9714bec9fb97 was (Author: maguowei): https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12713=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=f5211ead-5e53-5af8-f827-4dbf08df26bb > StreamPandasConversionTests Fails > - > > Key: FLINK-21216 > URL: https://issues.apache.org/jira/browse/FLINK-21216 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.12.0, 1.13.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12699=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3] > > === FAILURES > === > ___ StreamPandasConversionTests.test_empty_to_pandas > ___ > > self = > testMethod=test_empty_to_pandas> > > def test_empty_to_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type) > > pyflink/table/tests/test_pandas_conversion.py:144: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > pyflink/table/table_environment.py:1462: in from_pandas > arrow_schema = pa.Schema.from_pandas(pdf, preserve_index=False) > pyarrow/types.pxi:1315: in pyarrow.lib.Schema.from_pandas > ??? > .tox/py37-cython/lib/python3.7/site-packages/pyarrow/pandas_compat.py:519: in > dataframe_to_types > type_ = pa.lib._ndarray_to_arrow_type(values, type_) > pyarrow/array.pxi:53: in pyarrow.lib._ndarray_to_arrow_type > ??? > pyarrow/array.pxi:64: in pyarrow.lib._ndarray_to_type > ??? > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > > > ??? > E pyarrow.lib.ArrowTypeError: Did not pass numpy.dtype object > > pyarrow/error.pxi:108: ArrowTypeError > _ StreamPandasConversionTests.test_from_pandas > _ > > self = > testMethod=test_from_pandas> > > def test_from_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type, 5) > > pyflink/table/tests/test_pandas_conversion.py:120: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21203) Don’t collect -U&+U Row When they are equals In the LastRowFunction
[
https://issues.apache.org/jira/browse/FLINK-21203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276043#comment-17276043
]
Leonard Xu commented on FLINK-21203:
+1 for the ticket, [~wangpeibin]
About the config option, could you explain more?
> Don’t collect -U&+U Row When they are equals In the LastRowFunction
> -
>
> Key: FLINK-21203
> URL: https://issues.apache.org/jira/browse/FLINK-21203
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: wangpeibin
>Assignee: wangpeibin
>Priority: Major
>
> In the LastRowFunction , the -U&+U Row will be collected even if they are
> the same, which will increase calculation pressure of the next Operator.
>
> To avoid this, we can optimize the logic of DeduplicateFunctionHelper. Also,
> a config to enable the optimization will be added.
> With the sql followed:
> {quote}select * from
> (select
> *,
> row_number() over (partition by k order by proctime() desc ) as row_num
> from a
> ) t
> where row_num = 1
> {quote}
> Then input 2 row such as :
> {quote}Event("B","1","b"),
> Event("B","1","b")
> {quote}
> Now the output is:
> {quote}(true,+I[B, 1, b, 1])
> (false,-U[B, 1, b, 1])
> (true,+U[B, 1, b, 1])
> {quote}
> After the optimization, the output will be:
> {quote}(true,+I[B, 1, b, 1])
> {quote}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21216) StreamPandasConversionTests Fails
[ https://issues.apache.org/jira/browse/FLINK-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276042#comment-17276042 ] Guowei Ma commented on FLINK-21216: --- https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12713=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=f5211ead-5e53-5af8-f827-4dbf08df26bb > StreamPandasConversionTests Fails > - > > Key: FLINK-21216 > URL: https://issues.apache.org/jira/browse/FLINK-21216 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.12.0, 1.13.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12699=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3] > > === FAILURES > === > ___ StreamPandasConversionTests.test_empty_to_pandas > ___ > > self = > testMethod=test_empty_to_pandas> > > def test_empty_to_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type) > > pyflink/table/tests/test_pandas_conversion.py:144: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > pyflink/table/table_environment.py:1462: in from_pandas > arrow_schema = pa.Schema.from_pandas(pdf, preserve_index=False) > pyarrow/types.pxi:1315: in pyarrow.lib.Schema.from_pandas > ??? > .tox/py37-cython/lib/python3.7/site-packages/pyarrow/pandas_compat.py:519: in > dataframe_to_types > type_ = pa.lib._ndarray_to_arrow_type(values, type_) > pyarrow/array.pxi:53: in pyarrow.lib._ndarray_to_arrow_type > ??? > pyarrow/array.pxi:64: in pyarrow.lib._ndarray_to_type > ??? > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > > > ??? > E pyarrow.lib.ArrowTypeError: Did not pass numpy.dtype object > > pyarrow/error.pxi:108: ArrowTypeError > _ StreamPandasConversionTests.test_from_pandas > _ > > self = > testMethod=test_from_pandas> > > def test_from_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type, 5) > > pyflink/table/tests/test_pandas_conversion.py:120: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21215) UnalignedCheckpointITCase.execute Failed
[ https://issues.apache.org/jira/browse/FLINK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276041#comment-17276041 ] Guowei Ma commented on FLINK-21215: --- https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12713=logs=02c4e775-43bf-5625-d1cc-542b5209e072=e5961b24-88d9-5c77-efd3-955422674c25 > UnalignedCheckpointITCase.execute Failed > > > Key: FLINK-21215 > URL: https://issues.apache.org/jira/browse/FLINK-21215 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.13.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12691=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9146] > > > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:98) > > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:84) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1930) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1007) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$9(SchedulerBase.java:1009) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$processCheckpointCoordinatorMessage$10(SchedulerBase.java:1025) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * fe9b0d8824da2e50712f599978a4586bfe8f28a4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12712) * e97b4de046ec8231040f2c88b851a3fd69bf647b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12720) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
KarmaGYZ commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r567546171
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
##
@@ -70,6 +71,17 @@ public static WorkerResourceSpec fromTaskExecutorProcessSpec(
taskExecutorProcessSpec.getManagedMemorySize());
}
+public static WorkerResourceSpec fromTotalResourceProfile(
+final ResourceProfile resourceProfile) {
+Preconditions.checkNotNull(resourceProfile);
+return new WorkerResourceSpec(
+(CPUResource) resourceProfile.getCpuCores(),
Review comment:
+1, create https://issues.apache.org/jira/browse/FLINK-21220 for it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-21220) Change the type of cpuCores in ResourceProfile to CPUResource
Yangze Guo created FLINK-21220:
--
Summary: Change the type of cpuCores in ResourceProfile to
CPUResource
Key: FLINK-21220
URL: https://issues.apache.org/jira/browse/FLINK-21220
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Reporter: Yangze Guo
The {{cpuCores}} field in {{ResourceProfile}} should always be the CPUResource.
There is no need to generalize it to {{Resource}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] guoweiM commented on a change in pull request #14734: [FLINK-21066][runtime][checkpoint] Refactor CheckpointCoordinator to compute tasks to trigger/ack/commit dynamically
guoweiM commented on a change in pull request #14734:
URL: https://github.com/apache/flink/pull/14734#discussion_r567541421
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -563,7 +554,7 @@ private void
startTriggeringCheckpoint(CheckpointTriggerRequest request) {
createPendingCheckpoint(
timestamp,
request.props,
-ackTasks,
+checkpointBrief,
Review comment:
What I understand is that the `CheckpointBrief` describes how to “get” a
`CompleteChekcpoint`. There are some methods related to“get”a
CompleteCheckpoint, such as
“createPendingCheckpoint/reportToStateTracker/sendAbortedMessage/sendAcknowledgeMessages”.
However, these methods sometime use `CheckpointBrief`, sometime uses
`tasksTowaitFor/tasksToTriggerFor`, which are the internal state of the
`CheckpointCoordinator`. After we refactor maybe all the methods could only use
the `CheckpointBrief` as their parameter. (I think CheckpointId,TimeStamps
should also be in the CheckpointBrief).
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] guoweiM commented on a change in pull request #14734: [FLINK-21066][runtime][checkpoint] Refactor CheckpointCoordinator to compute tasks to trigger/ack/commit dynamically
guoweiM commented on a change in pull request #14734:
URL: https://github.com/apache/flink/pull/14734#discussion_r567528797
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointBriefCalculator.java
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.Execution;
+import org.apache.flink.runtime.executiongraph.ExecutionAttemptID;
+import org.apache.flink.runtime.executiongraph.ExecutionVertex;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** Computes the tasks to trigger, wait or commit for each checkpoint. */
+public class CheckpointBriefCalculator {
+private static final Logger LOG =
LoggerFactory.getLogger(CheckpointBriefCalculator.class);
+
+private final JobID jobId;
+
+private final List tasksToTrigger;
+
+private final List tasksToWait;
+
+private final List tasksToCommitTo;
+
+public CheckpointBriefCalculator(
+JobID jobId,
+List tasksToTrigger,
+List tasksToWait,
+List tasksToCommitTo) {
+
+this.jobId = jobId;
+this.tasksToTrigger = Collections.unmodifiableList(tasksToTrigger);
+this.tasksToWait = Collections.unmodifiableList(tasksToWait);
+this.tasksToCommitTo = Collections.unmodifiableList(tasksToCommitTo);
+}
+
+public CheckpointBrief calculateCheckpointBrief() throws
CheckpointException {
+List tasksToTrigger = getTriggerExecutions();
+Map tasksToWait = getAckTasks();
+
+return new CheckpointBrief(
+Collections.unmodifiableList(tasksToTrigger), tasksToWait,
tasksToCommitTo);
Review comment:
Maybe the `tasksToWait` could also be `unmodifialbeMap`
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -2122,7 +2065,7 @@ public boolean isForce() {
private Optional getVertex(ExecutionAttemptID id) throws
CheckpointException {
Review comment:
no `CheckpointException` any more.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -563,7 +554,7 @@ private void
startTriggeringCheckpoint(CheckpointTriggerRequest request) {
createPendingCheckpoint(
timestamp,
request.props,
-ackTasks,
+checkpointBrief,
Review comment:
What I understand is that the `CheckpointBrief` describes how to “get” a
`CompleteChekcpoint`. There are some methods related to“get”a
CompleteCheckpoint, such as
“createPendingCheckpoint/reportToStateTracker/sendAbortedMessage/sendAcknowledgeMessages”.
However, these methods sometime use `CheckpointBrief`, sometime uses
`tasksTowaitFor/tasksToTriggerFor`, which are the internal state of the
`CheckpointCoordinator`. After we refactor maybe all the methods could only use
the `CheckpointBrief` as their parameter. (I think CheckpointId,TimeStamps
should also be in the CheckpointBrief).
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -1843,6 +1843,7 @@ int getNumQueuedRequests() {
public void reportStats(long id, ExecutionAttemptID attemptId,
CheckpointMetrics metrics)
throws CheckpointException {
+
Review comment:
maybe we could remove this line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service,
[jira] [Updated] (FLINK-21207) 'csv.disable-quote-character' can not take effect during deserialization
[
https://issues.apache.org/jira/browse/FLINK-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-21207:
Description:
source table with 'csv.disable-quote-character' = 'true' can not take effect.
{code}
Caused by:
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Missing closing quote for valueCaused by:
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Missing closing quote for value at [Source: UNKNOWN; line: 1, column: 29] at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._reportParsingError(CsvParser.java:1250)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder._nextQuotedString(CsvDecoder.java:785)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder.nextString(CsvDecoder.java:630)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:846)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser.nextFieldName(CsvParser.java:665)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1282)
at
org.apache.flink.formats.csv.CsvRowDataDeserializationSchema.deserialize(CsvRowDataDeserializationSchema.java:155)
{code}
was:
source table with 'csv.disable-quote-character' = 'true' can not take effect.
Caused by:
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Missing closing quote for valueCaused by:
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Missing closing quote for value at [Source: UNKNOWN; line: 1, column: 29] at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._reportParsingError(CsvParser.java:1250)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder._nextQuotedString(CsvDecoder.java:785)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder.nextString(CsvDecoder.java:630)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:846)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser.nextFieldName(CsvParser.java:665)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1282)
at
org.apache.flink.formats.csv.CsvRowDataDeserializationSchema.deserialize(CsvRowDataDeserializationSchema.java:155)
> 'csv.disable-quote-character' can not take effect during deserialization
> -
>
> Key: FLINK-21207
> URL: https://issues.apache.org/jira/browse/FLINK-21207
> Project: Flink
> Issue Type: Bug
>Reporter: sharkd tu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.2, 1.13.0
>
>
> source table with 'csv.disable-quote-character' = 'true' can not take effect.
>
> {code}
> Caused by:
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
> Missing closing quote for valueCaused by:
>
[jira] [Assigned] (FLINK-21203) Don’t collect -U&+U Row When they are equals In the LastRowFunction
[
https://issues.apache.org/jira/browse/FLINK-21203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-21203:
---
Assignee: wangpeibin
> Don’t collect -U&+U Row When they are equals In the LastRowFunction
> -
>
> Key: FLINK-21203
> URL: https://issues.apache.org/jira/browse/FLINK-21203
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: wangpeibin
>Assignee: wangpeibin
>Priority: Major
>
> In the LastRowFunction , the -U&+U Row will be collected even if they are
> the same, which will increase calculation pressure of the next Operator.
>
> To avoid this, we can optimize the logic of DeduplicateFunctionHelper. Also,
> a config to enable the optimization will be added.
> With the sql followed:
> {quote}select * from
> (select
> *,
> row_number() over (partition by k order by proctime() desc ) as row_num
> from a
> ) t
> where row_num = 1
> {quote}
> Then input 2 row such as :
> {quote}Event("B","1","b"),
> Event("B","1","b")
> {quote}
> Now the output is:
> {quote}(true,+I[B, 1, b, 1])
> (false,-U[B, 1, b, 1])
> (true,+U[B, 1, b, 1])
> {quote}
> After the optimization, the output will be:
> {quote}(true,+I[B, 1, b, 1])
> {quote}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21203) Don’t collect -U&+U Row When they are equals In the LastRowFunction
[
https://issues.apache.org/jira/browse/FLINK-21203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276038#comment-17276038
]
Jark Wu commented on FLINK-21203:
-
Sounds good to me.
What do you think [~Leonard Xu]?
> Don’t collect -U&+U Row When they are equals In the LastRowFunction
> -
>
> Key: FLINK-21203
> URL: https://issues.apache.org/jira/browse/FLINK-21203
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: wangpeibin
>Priority: Major
>
> In the LastRowFunction , the -U&+U Row will be collected even if they are
> the same, which will increase calculation pressure of the next Operator.
>
> To avoid this, we can optimize the logic of DeduplicateFunctionHelper. Also,
> a config to enable the optimization will be added.
> With the sql followed:
> {quote}select * from
> (select
> *,
> row_number() over (partition by k order by proctime() desc ) as row_num
> from a
> ) t
> where row_num = 1
> {quote}
> Then input 2 row such as :
> {quote}Event("B","1","b"),
> Event("B","1","b")
> {quote}
> Now the output is:
> {quote}(true,+I[B, 1, b, 1])
> (false,-U[B, 1, b, 1])
> (true,+U[B, 1, b, 1])
> {quote}
> After the optimization, the output will be:
> {quote}(true,+I[B, 1, b, 1])
> {quote}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18312) SavepointStatusHandler and StaticFileServerHandler not redirect
[ https://issues.apache.org/jira/browse/FLINK-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139087#comment-17139087 ] Yu Wang edited comment on FLINK-18312 at 2/1/21, 3:04 AM: -- [~chesnay], [~trohrmann] agree with you, it's better to move the cache layer behind the RPC layer. was (Author: lucentwong): [~chesnay], [~trohrmann] agree with you, it's better to move the cache layer behind the PRC layer. > SavepointStatusHandler and StaticFileServerHandler not redirect > > > Key: FLINK-18312 > URL: https://issues.apache.org/jira/browse/FLINK-18312 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.8.0, 1.9.0, 1.10.0 > Environment: 1. Deploy flink cluster in standlone mode on kubernetes > and use two Jobmanagers for HA. > 2. Deploy a kubernetes service for the two jobmanagers to provide a unified > url. >Reporter: Yu Wang >Priority: Major > > Savepoint: > 1. Deploy our flink cluster in standlone mode on kubernetes and use two > Jobmanagers for HA. > 2. Deploy a kubernetes service for the two jobmanagers to provide a unified > url. > 3. Send a savepoint trigger request to the leader Jobmanager. > 4. Query the savepoint status from leader Jobmanager, get correct response. > 5. Query the savepoint status from standby Jobmanager, the response will be > 404. > Jobmanager log: > 1. Query log from leader Jobmanager, get leader log. > 2. Query log from standby Jobmanager, get standby log. > > Both these two requests will be redirect to the leader in 1.7. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * fe9b0d8824da2e50712f599978a4586bfe8f28a4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12712) * e97b4de046ec8231040f2c88b851a3fd69bf647b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
HuangZhenQiu commented on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770523596 @xintongsong Thanks. Your changes are reasonable and needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14678: [FLINK-20833][runtime] Add pluggable failure listener in job manager
flinkbot edited a comment on pull request #14678: URL: https://github.com/apache/flink/pull/14678#issuecomment-761905721 ## CI report: * cade20e85b29ca63c51383dca04976c1d9801042 UNKNOWN * 11651466eb13c5c05d2cd0eed0ac08b9bf617185 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12704) * 12652d9c7829d2d26d41f61185cbd7ab8f4ad505 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12719) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14824: [release-1.12][FLINK-21169][kafka] flink-connector-base dependency should be scope compile
flinkbot edited a comment on pull request #14824: URL: https://github.com/apache/flink/pull/14824#issuecomment-770512589 ## CI report: * e434ce611d8c12cceb749d22ea24f3e6fb650aeb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12718) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16224) Refine Hadoop Delegation Token based testing framework
[ https://issues.apache.org/jira/browse/FLINK-16224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276033#comment-17276033 ] jackwangcs commented on FLINK-16224: Hi [~rongr] and [~mapohl], is there any progress on this ticket? > Refine Hadoop Delegation Token based testing framework > -- > > Key: FLINK-16224 > URL: https://issues.apache.org/jira/browse/FLINK-16224 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Rong Rong >Priority: Major > > Currently the SecureTestEnvironment doesn't support Hadoop delegation token, > which makes the E2E testing of delegation-token-based YARN application > impossible. > Propose to enhance the testing framework to support delegation token based > launch in YARN cluster -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21216) StreamPandasConversionTests Fails
[ https://issues.apache.org/jira/browse/FLINK-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276030#comment-17276030 ] Huang Xingbo commented on FLINK-21216: -- I will take a look asap. > StreamPandasConversionTests Fails > - > > Key: FLINK-21216 > URL: https://issues.apache.org/jira/browse/FLINK-21216 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.12.0, 1.13.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12699=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3] > > === FAILURES > === > ___ StreamPandasConversionTests.test_empty_to_pandas > ___ > > self = > testMethod=test_empty_to_pandas> > > def test_empty_to_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type) > > pyflink/table/tests/test_pandas_conversion.py:144: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > pyflink/table/table_environment.py:1462: in from_pandas > arrow_schema = pa.Schema.from_pandas(pdf, preserve_index=False) > pyarrow/types.pxi:1315: in pyarrow.lib.Schema.from_pandas > ??? > .tox/py37-cython/lib/python3.7/site-packages/pyarrow/pandas_compat.py:519: in > dataframe_to_types > type_ = pa.lib._ndarray_to_arrow_type(values, type_) > pyarrow/array.pxi:53: in pyarrow.lib._ndarray_to_arrow_type > ??? > pyarrow/array.pxi:64: in pyarrow.lib._ndarray_to_type > ??? > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > > > ??? > E pyarrow.lib.ArrowTypeError: Did not pass numpy.dtype object > > pyarrow/error.pxi:108: ArrowTypeError > _ StreamPandasConversionTests.test_from_pandas > _ > > self = > testMethod=test_from_pandas> > > def test_from_pandas(self): > > table = self.t_env.from_pandas(self.pdf, self.data_type, 5) > > pyflink/table/tests/test_pandas_conversion.py:120: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14824: [release-1.12][FLINK-21169][kafka] flink-connector-base dependency should be scope compile
flinkbot commented on pull request #14824: URL: https://github.com/apache/flink/pull/14824#issuecomment-770512589 ## CI report: * e434ce611d8c12cceb749d22ea24f3e6fb650aeb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14824: [release-1.12][FLINK-21169][kafka] flink-connector-base dependency should be scope compile
flinkbot commented on pull request #14824: URL: https://github.com/apache/flink/pull/14824#issuecomment-770502976 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e434ce611d8c12cceb749d22ea24f3e6fb650aeb (Mon Feb 01 01:35:44 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise opened a new pull request #14824: [release-1.12][FLINK-21169][kafka] flink-connector-base dependency should be scope compile
tweise opened a new pull request #14824: URL: https://github.com/apache/flink/pull/14824 Backport of https://github.com/apache/flink/pull/14783 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14678: [FLINK-20833][runtime] Add pluggable failure listener in job manager
flinkbot edited a comment on pull request #14678: URL: https://github.com/apache/flink/pull/14678#issuecomment-761905721 ## CI report: * cade20e85b29ca63c51383dca04976c1d9801042 UNKNOWN * 11651466eb13c5c05d2cd0eed0ac08b9bf617185 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12704) * 12652d9c7829d2d26d41f61185cbd7ab8f4ad505 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-21169) Kafka flink-connector-base dependency should be scope compile
[ https://issues.apache.org/jira/browse/FLINK-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved FLINK-21169. -- Fix Version/s: 1.13.0 Resolution: Fixed > Kafka flink-connector-base dependency should be scope compile > - > > Key: FLINK-21169 > URL: https://issues.apache.org/jira/browse/FLINK-21169 > Project: Flink > Issue Type: Bug >Affects Versions: 1.12.1 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > The dependency is marked provided and therefore missing from an application > using the connector. This dependency should be packaged with the application > w/o extra declaration by the user. > It appears that was also the intention, based on other usage in > flink-connector-files ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] tweise merged pull request #14783: [FLINK-21169][kafka] flink-connector-base dependency should be scope compile
tweise merged pull request #14783: URL: https://github.com/apache/flink/pull/14783 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise commented on pull request #14783: [FLINK-21169][kafka] flink-connector-base dependency should be scope compile
tweise commented on pull request #14783: URL: https://github.com/apache/flink/pull/14783#issuecomment-770499151 > Hello @tweise and @becketqin ~ I think putting flink-connector-base JAR under lib directory of Flink distribution might be a better choice. After eventually all connectors are migrated to the new source and sink API, almost every job will reference this module as long as the job uses any type of connector, which would be a kind of duplication among all Flink job JARs. @PatrickRen anything under the lib directory is hard to deal with for users that don't control the Flink distribution. I prefer dependencies under lib to be as slim as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guowei Ma updated FLINK-21214: -- Affects Version/s: 1.11.0 > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276021#comment-17276021 ] Guowei Ma commented on FLINK-21214: --- https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12714=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20 > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21219) FlinkKafkaConsumer ignores offset overrides for new topics when restoring from savepoint.
[
https://issues.apache.org/jira/browse/FLINK-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276020#comment-17276020
]
Dominik Wosiński commented on FLINK-21219:
--
I would like to work on that.
> FlinkKafkaConsumer ignores offset overrides for new topics when restoring
> from savepoint.
> -
>
> Key: FLINK-21219
> URL: https://issues.apache.org/jira/browse/FLINK-21219
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.1
>Reporter: Dominik Wosiński
>Priority: Major
>
> Currently, when FlinkKafkaConsumer is restored from savepoint, the following
> code will handle topics that do not have offsets committed (for example if a
> new topic was added):
> {noformat}
> if (restoredState != null) { for (KafkaTopicPartition partition :
> allPartitions) { if (!restoredState.containsKey(partition)) {
> restoredState.put(partition,
> KafkaTopicPartitionStateSentinel.EARLIEST_OFFSET); } }{noformat}
>
> So if we have a KafkaConsumer with topicPattern and the pattern is changed,
> new topis will always start from earliest offset, even if originally the
> setting was different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-10407) FLIP-159: Reactive mode
[ https://issues.apache.org/jira/browse/FLINK-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276016#comment-17276016 ] John Phelan commented on FLINK-10407: - hi [~rmetzger] [~trohrmann] is there some work that an experienced developer, who's relatively new to the project, could do to help out with this ticket? I think this is an awesome feature! > FLIP-159: Reactive mode > --- > > Key: FLINK-10407 > URL: https://issues.apache.org/jira/browse/FLINK-10407 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Robert Metzger >Priority: Major > Fix For: 1.13.0 > > > The reactive mode is a new operation mode where a Flink cluster will react to > newly available resources (e.g. started by an external service) and make use > of them by rescaling the existing job. This ticket acts as an umbrella for > the required sub-tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21219) FlinkKafkaConsumer ignores offset overrides for new topics when restoring from savepoint.
Dominik Wosiński created FLINK-21219:
Summary: FlinkKafkaConsumer ignores offset overrides for new
topics when restoring from savepoint.
Key: FLINK-21219
URL: https://issues.apache.org/jira/browse/FLINK-21219
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.12.1
Reporter: Dominik Wosiński
Currently, when FlinkKafkaConsumer is restored from savepoint, the following
code will handle topics that do not have offsets committed (for example if a
new topic was added):
{noformat}
if (restoredState != null) { for (KafkaTopicPartition partition :
allPartitions) { if (!restoredState.containsKey(partition)) {
restoredState.put(partition, KafkaTopicPartitionStateSentinel.EARLIEST_OFFSET);
} }{noformat}
So if we have a KafkaConsumer with topicPattern and the pattern is changed, new
topis will always start from earliest offset, even if originally the setting
was different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * e7dbab13079f1a1f2350320930cc216c8ed42759 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12702) * dfd5c341320f3ccb7291c5833ae52a3d3f1632f1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12716) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * e7dbab13079f1a1f2350320930cc216c8ed42759 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12702) * dfd5c341320f3ccb7291c5833ae52a3d3f1632f1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * fe9b0d8824da2e50712f599978a4586bfe8f28a4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12712) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-20417) Handle "Too old resource version" exception in Kubernetes watch more gracefully
[
https://issues.apache.org/jira/browse/FLINK-20417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276004#comment-17276004
]
Emilien Kenler edited comment on FLINK-20417 at 1/31/21, 10:37 PM:
---
This issue happens event when the APIServer is not restarted.
We are running Kubernetes on Amazon EKS, and we have this exception about once
per hour, causing our job managers to restart.
{{2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in ResourceManager.2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in
ResourceManager.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]2021-01-31
21:57:54,528 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint
[] - Fatal error occurred in the cluster
entrypoint.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]}}
was (Author: milk):
This issue happens event when the APIServer is not restarted.
We are running Kubernetes on Amazon EKS, and we have this exception about once
per hour, causing our job managers to restart.
{{2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in ResourceManager.2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in
ResourceManager.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
[jira] [Commented] (FLINK-20417) Handle "Too old resource version" exception in Kubernetes watch more gracefully
[
https://issues.apache.org/jira/browse/FLINK-20417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276004#comment-17276004
]
Emilien Kenler commented on FLINK-20417:
This issue happens event when the APIServer is not restarted.
We are running Kubernetes on Amazon EKS, and we have this exception about once
per hour, causing our job managers to restart.
```
2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in ResourceManager.2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in
ResourceManager.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]2021-01-31
21:57:54,528 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint
[] - Fatal error occurred in the cluster
entrypoint.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
```
> Handle "Too old resource version" exception in Kubernetes watch more
> gracefully
> ---
>
> Key: FLINK-20417
> URL: https://issues.apache.org/jira/browse/FLINK-20417
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Affects Versions: 1.11.2, 1.12.0
>Reporter: Yang Wang
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, when the watcher(pods watcher, configmap watcher) is closed with
> exception, we will call {{WatchCallbackHandler#handleFatalError}}. And this
> could cause JobManager terminating and then failover.
> For most cases, this is correct. But not for "too old resource version"
> exception. See more information here[1]. Usually this exception could happen
> when the APIServer is restarted. And we just need to create a new watch and
> continue to do the pods/configmap watching. This could help the Flink cluster
> reducing
[jira] [Comment Edited] (FLINK-20417) Handle "Too old resource version" exception in Kubernetes watch more gracefully
[
https://issues.apache.org/jira/browse/FLINK-20417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276004#comment-17276004
]
Emilien Kenler edited comment on FLINK-20417 at 1/31/21, 10:36 PM:
---
This issue happens event when the APIServer is not restarted.
We are running Kubernetes on Amazon EKS, and we have this exception about once
per hour, causing our job managers to restart.
{{2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in ResourceManager.2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in
ResourceManager.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]2021-01-31
21:57:54,528 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint
[] - Fatal error occurred in the cluster
entrypoint.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]}}
was (Author: milk):
This issue happens event when the APIServer is not restarted.
We are running Kubernetes on Amazon EKS, and we have this exception about once
per hour, causing our job managers to restart.
```
2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in ResourceManager.2021-01-31 21:57:54,523 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in
ResourceManager.io.fabric8.kubernetes.client.KubernetesClientException: too old
resource version: 292832312 (294269953) at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
[flink-dist_2.12-1.12.1.jar:1.12.1] at
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * fe9b0d8824da2e50712f599978a4586bfe8f28a4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12712) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot edited a comment on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN * fe9b0d8824da2e50712f599978a4586bfe8f28a4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21139) ThresholdMeterTest.testMarkMultipleEvents unstable
[
https://issues.apache.org/jira/browse/FLINK-21139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275963#comment-17275963
]
Zhenqiu Huang commented on FLINK-21139:
---
[~xintongsong]
Make sense. Just created the PR. Please review it when you have time.
> ThresholdMeterTest.testMarkMultipleEvents unstable
> --
>
> Key: FLINK-21139
> URL: https://issues.apache.org/jira/browse/FLINK-21139
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Benchmarks, Build System, Tests
> Environment: $ java -version && javac -version
> openjdk version "11.0.8" 2020-07-14
> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10-202007292333)
> Eclipse OpenJ9 VM AdoptOpenJDK (build master-dc7cbe4bd, JRE 11 Linux
> riscv-64-Bit Compressed References 20200729_78 (JIT disabled, AOT disabled)
> OpenJ9 - dc7cbe4bd
> OMR - 1c0299f20
> JCL - 94b9d6d2c6 based on jdk-11.0.8+10)
> javac 11.0.8
> $ uname -a
> Linux test-gdams-debian10-riscv64-1.adoptopenjdk.net
> 5.0.0-rc1-56210-g0a657e0d72f0 #1 SMP Fri May 15 18:05:26 EDT 2020 riscv64
> GNU/Linux
> $ cat /proc/cpuinfo
> processor : 0
> hart : 1
> isa : rv64imafdc
> mmu : sv39
> uarch : sifive,rocket0
>Reporter: Ivan Serdyuk
>Assignee: Zhenqiu Huang
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> The test {{ThresholdMeterTest.testMarkMultipleEvents}} seems to be unstable.
> The version of Flink is
> ??$ git log -1??
> ??commit 6e77cfdff8a358adab4ab770a503197d95a64440 (HEAD -> master,
> origin/master, origin/HEAD)??
> ??Author: Roman Khachatryan ??
> ??Date: Fri Jan 22 14:39:48 2021 +0100??
> The test fails with the following exception:
> {code}
> Expected: a numeric value within <1.0E-6> of <40.0>
> but: <0.0> differed by <39.99>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.metrics.ThresholdMeterTest.testMarkMultipleEvents(ThresholdMeterTest.java:58)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
> at
>
[GitHub] [flink] flinkbot commented on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot commented on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770437030 ## CI report: * f0ba4a5828295c651491d7b031dd6c894375234b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14823: [FLINK-21139][Build] Fix unstable ThresholdMeterTest
flinkbot commented on pull request #14823: URL: https://github.com/apache/flink/pull/14823#issuecomment-770435644 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f0ba4a5828295c651491d7b031dd6c894375234b (Sun Jan 31 19:22:45 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
