[GitHub] [flink] KarmaGYZ commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
KarmaGYZ commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r565790739
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/PendingTaskManagerId.java
##
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager.slotmanager;
+
+import org.apache.flink.util.AbstractID;
+
+/** Id of {@link PendingTaskManager}. */
+public class PendingTaskManagerId extends AbstractID {
+private PendingTaskManagerId() {}
Review comment:
Why we need that as A call to the parent class's empty constructor
super() is done automatically?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14783: [FLINK-21169][kafka] flink-connector-base dependency should be scope compile
flinkbot commented on pull request #14783: URL: https://github.com/apache/flink/pull/14783#issuecomment-768765175 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ea0fa2c3a97b2d2082b40732850ccc960ba7a09e (Thu Jan 28 03:04:31 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21169) Kafka flink-connector-base dependency should be scope compile
[ https://issues.apache.org/jira/browse/FLINK-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21169: --- Labels: pull-request-available (was: ) > Kafka flink-connector-base dependency should be scope compile > - > > Key: FLINK-21169 > URL: https://issues.apache.org/jira/browse/FLINK-21169 > Project: Flink > Issue Type: Bug >Affects Versions: 1.12.1 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > > The dependency is marked provided and therefore missing from an application > using the connector. This dependency should be packaged with the application > w/o extra declaration by the user. > It appears that was also the intention, based on other usage in > flink-connector-files ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] tweise opened a new pull request #14783: [FLINK-21169][kafka] flink-connector-base dependency should be scope compile
tweise opened a new pull request #14783: URL: https://github.com/apache/flink/pull/14783 ## What is the purpose of the change Changes the scope of `flink-connector-base` dependency from `provided` to default so that the dependency becomes part of application classpath. Otherwise, it needs to be repeated downstream to be able to use the connector. ## Verifying this change Verified effect of the change by running application within IDE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21169) Kafka flink-connector-base dependency should be scope compile
Thomas Weise created FLINK-21169: Summary: Kafka flink-connector-base dependency should be scope compile Key: FLINK-21169 URL: https://issues.apache.org/jira/browse/FLINK-21169 Project: Flink Issue Type: Bug Affects Versions: 1.12.1 Reporter: Thomas Weise Assignee: Thomas Weise The dependency is marked provided and therefore missing from an application using the connector. This dependency should be packaged with the application w/o extra declaration by the user. It appears that was also the intention, based on other usage in flink-connector-files ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21005) Introduce new provider for unified Sink API and implement in planner
[
https://issues.apache.org/jira/browse/FLINK-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272539#comment-17272539
]
Nicholas Jiang edited comment on FLINK-21005 at 1/28/21, 2:37 AM:
--
[~benjobs], if you aren't available for this issue, I would like to work for
this issue.
[~jark], if [~benjobs] has no response for the issue, could you please assign
this to me?
was (Author: nicholasjiang):
[~benjobs], if you aren't available for this issue, I would like to work for
this issue.
> Introduce new provider for unified Sink API and implement in planner
>
>
> Key: FLINK-21005
> URL: https://issues.apache.org/jira/browse/FLINK-21005
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Huajie Wang
>Priority: Major
> Fix For: 1.13.0
>
>
> FLIP-143 [1] introduced the unified sink API, we should add a
> {{SinkRuntimeProvider}} for it and support it in planner. So that Table SQL
> users can also use the unified sink APIs.
> [1]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21168) Add support for general python group window aggregate function in Physical Rule and Node
Huang Xingbo created FLINK-21168: Summary: Add support for general python group window aggregate function in Physical Rule and Node Key: FLINK-21168 URL: https://issues.apache.org/jira/browse/FLINK-21168 Project: Flink Issue Type: Sub-task Components: API / Python, Table SQL / Planner Affects Versions: 1.13.0 Reporter: Huang Xingbo Fix For: 1.13.0 Add support for general python group window aggregate function in Physical Rule and Node -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21154) Move the shared request-reply protobuf files to a separate artifact.
[ https://issues.apache.org/jira/browse/FLINK-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-21154. --- Fix Version/s: statefun-3.0.0 Resolution: Fixed flink-statefun/master: 24da73beb6d38d93c007c1c541ef922c1de60c43 > Move the shared request-reply protobuf files to a separate artifact. > > > Key: FLINK-21154 > URL: https://issues.apache.org/jira/browse/FLINK-21154 > Project: Flink > Issue Type: Task > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.0.0 > > > Currently the request-reply.proto file is located in the statefun-flink-core > artifact. This file is also used to manually generate the Python SDK > request_reply_pb2.py file. Since we are about to add additional SDKS and > expand the number of proto files, generating these by hand can get quickly > out of hand. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai closed pull request #194: [FLINK-21154] Move the shared request-reply Protobuf files to a separate artifact
tzulitai closed pull request #194: URL: https://github.com/apache/flink-statefun/pull/194 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
KarmaGYZ commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r565777058
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/FineGrainedTaskManagerSlot.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager.slotmanager;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotID;
+import org.apache.flink.runtime.instance.InstanceID;
+import
org.apache.flink.runtime.resourcemanager.registration.TaskExecutorConnection;
+import org.apache.flink.util.Preconditions;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A FineGrainedTaskManagerSlot represents a slot located in a TaskManager. It
contains the
+ * necessary information for the allocation of the slot, and keeps track of
the state of the slot.
+ * Note that it should in the state of {@link SlotState#FREE}.
Review comment:
nice catch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14774: [FLINK-21163][python] Fix the issue that Python dependencies specified via CLI override the dependencies specified in configuration
flinkbot edited a comment on pull request #14774: URL: https://github.com/apache/flink/pull/14774#issuecomment-768234538 ## CI report: * 25e36a813527d64dab09dde6c13cbdab78bf05ab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12558) * 54a8354c03402cedbfe56f8e8e7336f2d9072e34 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12575) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
KarmaGYZ commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r565769920
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/FineGrainedTaskManagerRegistration.java
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager.slotmanager;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.instance.InstanceID;
+import
org.apache.flink.runtime.resourcemanager.registration.TaskExecutorConnection;
+import org.apache.flink.util.Preconditions;
+
+import java.math.BigDecimal;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A FineGrainedTaskManagerRegistration represents a TaskManager. It records
the internal state of
+ * the TaskManager, including allocated/pending slots, total/available
resources.
+ *
+ * This class is the fine-grained resource management version of the {@link
+ * TaskManagerRegistration}.
+ */
+public class FineGrainedTaskManagerRegistration implements TaskManagerInfo {
+private final TaskExecutorConnection taskManagerConnection;
+
+private final Map slots;
+
+private final ResourceProfile defaultSlotResourceProfile;
+
+private final ResourceProfile totalResource;
+
+private final int defaultNumSlots;
+
+private ResourceProfile unusedResource;
+
+private ResourceProfile pendingResource = ResourceProfile.ZERO;
+
+/** Timestamp when the last time becoming idle. Otherwise Long.MAX_VALUE.
*/
+private long idleSince;
+
+public FineGrainedTaskManagerRegistration(
+TaskExecutorConnection taskManagerConnection,
+ResourceProfile totalResourceProfile,
+ResourceProfile defaultSlotResourceProfile) {
+this.taskManagerConnection =
Preconditions.checkNotNull(taskManagerConnection);
+this.totalResource = Preconditions.checkNotNull(totalResourceProfile);
+this.defaultSlotResourceProfile =
Preconditions.checkNotNull(defaultSlotResourceProfile);
+
+this.slots = new HashMap<>(16);
+
+this.defaultNumSlots =
+totalResourceProfile
+.getCpuCores()
+.getValue()
+.divide(
+
defaultSlotResourceProfile.getCpuCores().getValue(),
+0,
+BigDecimal.ROUND_DOWN)
+.intValue();
Review comment:
Yes, I also agree that the number of registered/free slots might be
invalid under fine-grained resource management.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #14779: [FLINK-21158][Runtime/Web Frontend] wrong jvm metaspace and overhead size show in taskmanager metric page
xintongsong commented on pull request #14779: URL: https://github.com/apache/flink/pull/14779#issuecomment-768742573 @pengkangjing Please take a look at the CI report. There's a compiling failure due to code style violation. Please refer to the following guidelines on how to use the new code format tool. https://ci.apache.org/projects/flink/flink-docs-release-1.12/flinkDev/ide_setup.html#code-formatting This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14774: [FLINK-21163][python] Fix the issue that Python dependencies specified via CLI override the dependencies specified in configuration
flinkbot edited a comment on pull request #14774: URL: https://github.com/apache/flink/pull/14774#issuecomment-768234538 ## CI report: * 25e36a813527d64dab09dde6c13cbdab78bf05ab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12558) * 54a8354c03402cedbfe56f8e8e7336f2d9072e34 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #14774: [FLINK-21163][python] Fix the issue that Python dependencies specified via CLI override the dependencies specified in configuration
dianfu commented on pull request #14774: URL: https://github.com/apache/flink/pull/14774#issuecomment-768719211 @WeiZhong94 Thanks a lot for the view. Updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
xintongsong commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r565154346
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/FineGrainedTaskManagerTrackerTest.java
##
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager.slotmanager;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import
org.apache.flink.runtime.resourcemanager.registration.TaskExecutorConnection;
+import org.apache.flink.runtime.slots.ResourceCounter;
+import org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGatewayBuilder;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import java.util.Collections;
+
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for the {@link FineGrainedTaskManagerTracker}. */
+public class FineGrainedTaskManagerTrackerTest extends TestLogger {
Review comment:
I think we might need to cover more cases.
- When `recordPendingAllocations` is called multiple times, only the last
time is recorded.
- `removePendingTaskManager` called on pending task manager where pending
slots are allocated.
And also some failure cases.
- Trying to remove a (pending) task manager that does not exist.
- Try to allocate slot from a (pending) task manager that does not have
enough resource
- Try to free a non-exist slot.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-20663) Managed memory may not be released in time when operators use managed memory frequently
[
https://issues.apache.org/jira/browse/FLINK-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273247#comment-17273247
]
Kurt Young commented on FLINK-20663:
IIRC in the past, users of managed memory (operators, data structures) are
required to know the contract when using memory segment, which is when they
release the memory segments, they can just return the object back to memory
manager, and don't access it ever again.
And {{HybridMemorySegment}} and {{MemorySegment also provided some protection
after being released in their free() method. The only exception is wrap()
method, which caused FLINK-14894. But wrap() method is not that popular, and
half of them are used by network buffer which is not even managed (they are
unpooled segments). Other than this, all other methods are safe to use and will
raise exception if being misused. }}
So my thought is, instead of relying on such inefficient and complex way (Full
GC) to protect only one usage about memory segment, we can change this to much
simpler way (like the old behaviors) and we pay enough attention of the wrap()
method, e.g. adding some comments or find some way to protect the wrapped
ByteBuffer.
> Managed memory may not be released in time when operators use managed memory
> frequently
> ---
>
> Key: FLINK-20663
> URL: https://issues.apache.org/jira/browse/FLINK-20663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.12.2
>
>
> Some batch operators (like sort merge join or hash aggregate) use managed
> memory frequently. When these operators are chained together and the cluster
> load is a bit heavy, it is very likely that the following exception occurs:
> {code:java}
> 2020-12-18 10:04:32
> java.lang.RuntimeException:
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate
> 512 pages
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:85)
> at
> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.(SimpleCollectingOutputView.java:49)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.(BytesHashMap.java:297)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:103)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:90)
> at LocalHashAggregateWithKeys$209161.open(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:506)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:834)
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209161.close(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.TableStreamOperator.dispose(TableStreamOperator.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:739)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:719)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:642)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:551)
> ... 3 more
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209766.close(Unknown
> Source)
> ... 8 more
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 512 pages
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:231)
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:83)
> ... 13 more
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 16777216 bytes, only 9961487 bytes are remaining. This usually
> indicates that yo
[GitHub] [flink] KarmaGYZ commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
KarmaGYZ commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r565757084
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/PendingTaskManagerId.java
##
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager.slotmanager;
+
+import org.apache.flink.util.AbstractID;
+
+/** Id of {@link PendingTaskManager}. */
+public class PendingTaskManagerId extends AbstractID {
Review comment:
Not sure whether we need it. This ID should only be used inside the
slotmanager. It will not be serialized in any case, and thus no need to check
the version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14782: [FLINK-21167] Make StateTable snapshots iterable
flinkbot edited a comment on pull request #14782: URL: https://github.com/apache/flink/pull/14782#issuecomment-768559542 ## CI report: * d3fa4291770bf08966eafb0ca2e4d7f3ffe0912a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12570) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.
[ https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273212#comment-17273212 ] Xiaoguang Sun edited comment on FLINK-21108 at 1/28/21, 12:24 AM: -- Cluster IP can still be accessed by anybody running within the cluster. We can use SSL authentication for ingress at this time, this seems to be the only way to make sure only ingress can access Flink console. So how about rest client side authentication, it has to be done to make sure validated users can submit job to Flink. was (Author: xexplorer): Cluster IP can be accessed by anybody running within the cluster. We can use SSL authentication for ingress at this time, this seems to be the only way to make sure only ingress can access Flink console. So how about rest client side authentication, it has to be done to make sure validated users can submit job to Flink. > Flink runtime rest server and history server webmonitor do not require > authentication. > -- > > Key: FLINK-21108 > URL: https://issues.apache.org/jira/browse/FLINK-21108 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Xiaoguang Sun >Priority: Major > Labels: pull-request-available > > Flink runtime rest server and history server webmonitor do not require > authentication. At certain scenarios, prohibiting unauthorized access is > desired. Http basic authentication can be used here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.
[ https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273212#comment-17273212 ] Xiaoguang Sun commented on FLINK-21108: --- Cluster IP can be accessed by anybody running within the cluster. We can use SSL authentication for ingress at this time, this seems to be the only way to make sure only ingress can access Flink console. So how about rest client side authentication, it has to be done to make sure validated users can submit job to Flink. > Flink runtime rest server and history server webmonitor do not require > authentication. > -- > > Key: FLINK-21108 > URL: https://issues.apache.org/jira/browse/FLINK-21108 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Xiaoguang Sun >Priority: Major > Labels: pull-request-available > > Flink runtime rest server and history server webmonitor do not require > authentication. At certain scenarios, prohibiting unauthorized access is > desired. Http basic authentication can be used here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman closed pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
sjwiesman closed pull request #13912: URL: https://github.com/apache/flink/pull/13912 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21059) KafkaSourceEnumerator does not honor consumer properties
[ https://issues.apache.org/jira/browse/FLINK-21059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273170#comment-17273170 ] Thomas Weise commented on FLINK-21059: -- 1.12: https://github.com/apache/flink/commit/1822d138016c6efddaf98b73de420ddedb5c6fd6 > KafkaSourceEnumerator does not honor consumer properties > > > Key: FLINK-21059 > URL: https://issues.apache.org/jira/browse/FLINK-21059 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > > Enumerator fails when SSL is required because the user provided properties > are not used to construct the admin client. Empty properties are created and > all provided properties except bootstrap.servers are ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-21059) KafkaSourceEnumerator does not honor consumer properties
[ https://issues.apache.org/jira/browse/FLINK-21059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved FLINK-21059. -- Resolution: Fixed > KafkaSourceEnumerator does not honor consumer properties > > > Key: FLINK-21059 > URL: https://issues.apache.org/jira/browse/FLINK-21059 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0, 1.12.2 > > > Enumerator fails when SSL is required because the user provided properties > are not used to construct the admin client. Empty properties are created and > all provided properties except bootstrap.servers are ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21059) KafkaSourceEnumerator does not honor consumer properties
[ https://issues.apache.org/jira/browse/FLINK-21059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated FLINK-21059: - Fix Version/s: 1.12.2 1.13.0 > KafkaSourceEnumerator does not honor consumer properties > > > Key: FLINK-21059 > URL: https://issues.apache.org/jira/browse/FLINK-21059 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.13.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0, 1.12.2 > > > Enumerator fails when SSL is required because the user provided properties > are not used to construct the admin client. Empty properties are created and > all provided properties except bootstrap.servers are ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] tweise merged pull request #14781: [release-1.12][FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
tweise merged pull request #14781: URL: https://github.com/apache/flink/pull/14781 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14781: [release-1.12][FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
flinkbot edited a comment on pull request #14781: URL: https://github.com/apache/flink/pull/14781#issuecomment-768462281 ## CI report: * c5896f26c6106ab25c7c9c24bf9867dabca83e98 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12569) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14782: [FLINK-21167] Make StateTable snapshots iterable
flinkbot edited a comment on pull request #14782: URL: https://github.com/apache/flink/pull/14782#issuecomment-768559542 ## CI report: * d3fa4291770bf08966eafb0ca2e4d7f3ffe0912a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12570) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14782: [FLINK-21167] Make StateTable snapshots iterable
flinkbot commented on pull request #14782: URL: https://github.com/apache/flink/pull/14782#issuecomment-768559542 ## CI report: * d3fa4291770bf08966eafb0ca2e4d7f3ffe0912a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * 3885663038156d52aec21062a9743009fb875cc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12535) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14782: [FLINK-21167] Make StateTable snapshots iterable
flinkbot commented on pull request #14782: URL: https://github.com/apache/flink/pull/14782#issuecomment-768547756 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d3fa4291770bf08966eafb0ca2e4d7f3ffe0912a (Wed Jan 27 20:12:50 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21167) Make StateTable snapshots iterable
[
https://issues.apache.org/jira/browse/FLINK-21167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-21167:
---
Labels: pull-request-available (was: )
> Make StateTable snapshots iterable
> --
>
> Key: FLINK-21167
> URL: https://issues.apache.org/jira/browse/FLINK-21167
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing, Runtime / State Backends
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> In order to implement an iterator required by a binary unified savepoint we
> need a way to iterate a snapshot. We should add an interface
> {code}
> @Internal
> public interface IterableStateSnapshot extends StateSnapshot {
> Iterator> getIterator(int keyGroup);
> }
> {code}
> that will be implemented by StateTableSnapshots.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dawidwys opened a new pull request #14782: [FLINK-21167] Make StateTable snapshots iterable
dawidwys opened a new pull request #14782: URL: https://github.com/apache/flink/pull/14782 ## What is the purpose of the change In order to implement an iterator required by a binary unified savepoint we need a way to iterate a snapshot. ## Verifying this change Added tests in: * CopyOnWriteStateMapTest * NestedMapsStateTableTest * CopyOnWriteSkipListStateMapComplexOpTest ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**yes** / no): test dependencies - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**yes** / no / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * 3885663038156d52aec21062a9743009fb875cc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12535) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21167) Make StateTable snapshots iterable
Dawid Wysakowicz created FLINK-21167:
Summary: Make StateTable snapshots iterable
Key: FLINK-21167
URL: https://issues.apache.org/jira/browse/FLINK-21167
Project: Flink
Issue Type: Sub-task
Components: Runtime / Checkpointing, Runtime / State Backends
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
Fix For: 1.13.0
In order to implement an iterator required by a binary unified savepoint we
need a way to iterate a snapshot. We should add an interface
{code}
@Internal
public interface IterableStateSnapshot extends StateSnapshot {
Iterator> getIterator(int keyGroup);
}
{code}
that will be implemented by StateTableSnapshots.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] sjwiesman commented on a change in pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
sjwiesman commented on a change in pull request #14780:
URL: https://github.com/apache/flink/pull/14780#discussion_r565575632
##
File path: docs/flinkDev/building.md
##
@@ -132,7 +132,7 @@ Since version 1.7 Flink builds with Scala version 2.11
(default) and 2.12.
To build FLink against Scala 2.12, issue the following command:
{% highlight bash %}
-mvn clean install -DskipTests -Dscala-2.12
+mvn clean install -DskipTests -Dscala-2.12 -Dscala.version=2.12.12
Review comment:
Maybe put `-scala.version=`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
sjwiesman commented on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-768537882 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman removed a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
sjwiesman removed a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-768343868 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
sjwiesman commented on a change in pull request #14780:
URL: https://github.com/apache/flink/pull/14780#discussion_r565575632
##
File path: docs/flinkDev/building.md
##
@@ -132,7 +132,7 @@ Since version 1.7 Flink builds with Scala version 2.11
(default) and 2.12.
To build FLink against Scala 2.12, issue the following command:
{% highlight bash %}
-mvn clean install -DskipTests -Dscala-2.12
+mvn clean install -DskipTests -Dscala-2.12 -Dscala.version=2.12.12
Review comment:
Maybe put ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-21166) IllegalStateException in CheckpointMetricsBuilder surfacing in "Resuming Savepoint" e2e
[
https://issues.apache.org/jira/browse/FLINK-21166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-21166:
---
Summary: IllegalStateException in CheckpointMetricsBuilder surfacing in
"Resuming Savepoint" e2e (was: NullPointer in CheckpointMetricsBuilder
surfacing in "Resuming Savepoint" e2e)
> IllegalStateException in CheckpointMetricsBuilder surfacing in "Resuming
> Savepoint" e2e
> ---
>
> Key: FLINK-21166
> URL: https://issues.apache.org/jira/browse/FLINK-21166
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12562&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> Running 'Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test'
> Found exception in log files; printing first 500 lines; see full logs for
> details:
> ...
> [FAIL] 'Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test'
> failed after 0 minutes and 30 seconds! Test exited with exit code 0 but the
> logs contained errors, exceptions or non-empty .out files
> {code}
> One TaskManager log contains the following:
> {code}
> === Finished metrics report
> ===
> 2021-01-27 15:22:49,635 WARN
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Could
> not properly clean up the async checkpoint runnable.
> java.lang.IllegalStateException: null
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:177)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.util.Preconditions.checkCompletedNormally(Preconditions.java:261)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.concurrent.FutureUtils.checkStateAndGet(FutureUtils.java:1176)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder.build(CheckpointMetricsBuilder.java:133)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.reportAbortedSnapshotStats(AsyncCheckpointRunnable.java:219)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.close(AsyncCheckpointRunnable.java:292)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:275)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.cancelAsyncCheckpointRunnable(SubtaskCheckpointCoordinatorImpl.java:451)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointAborted(SubtaskCheckpointCoordinatorImpl.java:340)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointAbortAsync$12(StreamTask.java:1069)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1082)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:314)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:300)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:188)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:615)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:579)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task
[GitHub] [flink] sjwiesman edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
sjwiesman edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-768343868 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20663) Managed memory may not be released in time when operators use managed memory frequently
[
https://issues.apache.org/jira/browse/FLINK-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273080#comment-17273080
]
Till Rohrmann commented on FLINK-20663:
---
Thanks for reporting this issue. Which Java version are you using when this
problem occurs? Versions below jdk8u72 don't reliably work (FLINK-18581).
Can you reproduce the same problem when using the DataSet API? I just want to
make sure that we don't have a resource leak somewhere. Maybe drawing a heap
dump could help shine some light on the problem.
Just to give a bit of background: The problem FLINK-15758 is supposed to solve
is FLINK-14894. Since we don't do ref counting for the {{MemorySegments}} it is
hard to know when we can truly release a {{MemorySegment}}. If we release a
segment and we still have an owner which accesses this segment, then this can
potentially lead to memory corruption. It is best described here:
https://issues.apache.org/jira/browse/FLINK-14894?focusedCommentId=17023015&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17023015.
> Managed memory may not be released in time when operators use managed memory
> frequently
> ---
>
> Key: FLINK-20663
> URL: https://issues.apache.org/jira/browse/FLINK-20663
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.12.2
>
>
> Some batch operators (like sort merge join or hash aggregate) use managed
> memory frequently. When these operators are chained together and the cluster
> load is a bit heavy, it is very likely that the following exception occurs:
> {code:java}
> 2020-12-18 10:04:32
> java.lang.RuntimeException:
> org.apache.flink.runtime.memory.MemoryAllocationException: Could not allocate
> 512 pages
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:85)
> at
> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.(SimpleCollectingOutputView.java:49)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.(BytesHashMap.java:297)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:103)
> at
> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.(BytesHashMap.java:90)
> at LocalHashAggregateWithKeys$209161.open(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:401)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:506)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:834)
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209161.close(Unknown Source)
> at
> org.apache.flink.table.runtime.operators.TableStreamOperator.dispose(TableStreamOperator.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:739)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:719)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:642)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:551)
> ... 3 more
> Suppressed: java.lang.NullPointerException
> at LocalHashAggregateWithKeys$209766.close(Unknown
> Source)
> ... 8 more
> Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could
> not allocate 512 pages
> at
> org.apache.flink.runtime.memory.MemoryManager.allocatePages(MemoryManager.java:231)
> at
> org.apache.flink.table.runtime.util.LazyMemorySegmentPool.nextSegment(LazyMemorySegmentPool.java:83)
> ... 13 more
> Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could
> not allocate 16777216 bytes, only 9961487 bytes are remaining. This usually
> indicates that you are requesting more memory than you have reserved.
> However, when running an old JVM version it can also be ca
[jira] [Updated] (FLINK-21166) NullPointer in CheckpointMetricsBuilder surfacing in "Resuming Savepoint" e2e
[
https://issues.apache.org/jira/browse/FLINK-21166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-21166:
---
Fix Version/s: 1.13.0
> NullPointer in CheckpointMetricsBuilder surfacing in "Resuming Savepoint" e2e
> -
>
> Key: FLINK-21166
> URL: https://issues.apache.org/jira/browse/FLINK-21166
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12562&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> Running 'Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test'
> Found exception in log files; printing first 500 lines; see full logs for
> details:
> ...
> [FAIL] 'Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test'
> failed after 0 minutes and 30 seconds! Test exited with exit code 0 but the
> logs contained errors, exceptions or non-empty .out files
> {code}
> One TaskManager log contains the following:
> {code}
> === Finished metrics report
> ===
> 2021-01-27 15:22:49,635 WARN
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Could
> not properly clean up the async checkpoint runnable.
> java.lang.IllegalStateException: null
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:177)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.util.Preconditions.checkCompletedNormally(Preconditions.java:261)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.concurrent.FutureUtils.checkStateAndGet(FutureUtils.java:1176)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder.build(CheckpointMetricsBuilder.java:133)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.reportAbortedSnapshotStats(AsyncCheckpointRunnable.java:219)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.close(AsyncCheckpointRunnable.java:292)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:275)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.cancelAsyncCheckpointRunnable(SubtaskCheckpointCoordinatorImpl.java:451)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointAborted(SubtaskCheckpointCoordinatorImpl.java:340)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointAbortAsync$12(StreamTask.java:1069)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1082)
> ~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:314)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:300)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:188)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:615)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:579)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:565)
> [flink-dist_2.11-1.13-SNAPSHOT.jar:1
[GitHub] [flink] rmetzger commented on pull request #14767: [FLINK-21099] Introduce JobType to distinguish between batch and streaming jobs
rmetzger commented on pull request #14767: URL: https://github.com/apache/flink/pull/14767#issuecomment-768488566 Reported e2e test instability here: https://issues.apache.org/jira/browse/FLINK-21166 Otherwise, I've addressed all your comments! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21166) NullPointer in CheckpointMetricsBuilder surfacing in "Resuming Savepoint" e2e
Robert Metzger created FLINK-21166:
--
Summary: NullPointer in CheckpointMetricsBuilder surfacing in
"Resuming Savepoint" e2e
Key: FLINK-21166
URL: https://issues.apache.org/jira/browse/FLINK-21166
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.13.0
Reporter: Robert Metzger
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12562&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529
{code}
Running 'Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test'
Found exception in log files; printing first 500 lines; see full logs for
details:
...
[FAIL] 'Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test'
failed after 0 minutes and 30 seconds! Test exited with exit code 0 but the
logs contained errors, exceptions or non-empty .out files
{code}
One TaskManager log contains the following:
{code}
=== Finished metrics report ===
2021-01-27 15:22:49,635 WARN
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Could not
properly clean up the async checkpoint runnable.
java.lang.IllegalStateException: null
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:177)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.util.Preconditions.checkCompletedNormally(Preconditions.java:261)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.runtime.concurrent.FutureUtils.checkStateAndGet(FutureUtils.java:1176)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder.build(CheckpointMetricsBuilder.java:133)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.reportAbortedSnapshotStats(AsyncCheckpointRunnable.java:219)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.close(AsyncCheckpointRunnable.java:292)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:275)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.cancelAsyncCheckpointRunnable(SubtaskCheckpointCoordinatorImpl.java:451)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointAborted(SubtaskCheckpointCoordinatorImpl.java:340)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointAbortAsync$12(StreamTask.java:1069)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1082)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:314)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:300)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:188)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:615)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:579)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:565)
[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2021-01-27 15:22:49,637 WARN
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Could not
properly clean up the async checkpoint runnable.
java.lang.IllegalStateException: null
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:177)
~[flink-dist_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
at
org.apac
[GitHub] [flink] aljoscha commented on a change in pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
aljoscha commented on a change in pull request #14780:
URL: https://github.com/apache/flink/pull/14780#discussion_r565528105
##
File path: docs/flinkDev/building.md
##
@@ -132,7 +132,7 @@ Since version 1.7 Flink builds with Scala version 2.11
(default) and 2.12.
To build FLink against Scala 2.12, issue the following command:
{% highlight bash %}
-mvn clean install -DskipTests -Dscala-2.12
+mvn clean install -DskipTests -Dscala-2.12 -Dscala.version=2.12.12
Review comment:
This is to show that you can use a custom version, which is the purpose
of this PR. I think you only want to build it yourself to change the exact
Scala binary version. Otherwise, you can just use the binary releases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14781: [release-1.12][FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
flinkbot edited a comment on pull request #14781: URL: https://github.com/apache/flink/pull/14781#issuecomment-768462281 ## CI report: * c5896f26c6106ab25c7c9c24bf9867dabca83e98 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12569) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
flinkbot edited a comment on pull request #14765: URL: https://github.com/apache/flink/pull/14765#issuecomment-767648687 ## CI report: * 31d80dfb565af80b87945d5b717c6ea3d2b8d0ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12568) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14781: [release-1.12][FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
flinkbot commented on pull request #14781: URL: https://github.com/apache/flink/pull/14781#issuecomment-768462281 ## CI report: * c5896f26c6106ab25c7c9c24bf9867dabca83e98 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273052#comment-17273052
]
Kezhu Wang commented on FLINK-21132:
I did not find an easy/viable way to distinguish truly end of input from
stop-with-savepoint style end of input. They are both {{EndOfPartitionEvent}}
in network. Legacy sources could also emit elements between synchronous
savepoint and {{EndOfPartitionEvent}}.
{quote}that is when the source and all the predecessor operators have shut down.
{quote}
How this could be viable for, say, a no chained head operator ? AFAIK, no
chained operator has only two inputs from outside: rpc call through
{{TaskExecutorGateway}} and stream elements and events from predecessor
operators through input channel. I saw only {{EndOfPartitionEvent}} related.
I think stop-with-savepoint has enough room to operate on *all tasks* not just
*source tasks* without interfering with data flow upon
{{StreamTask.notifyCheckpointComplete}}. I could give an simple(probably not
ideal) solution, let {{Task}} query {{AbstractInvokable}} to know whether it
should *finish* partition writers after successful run. This way FLINK-21133 is
solved also.
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Priority: Major
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
flinkbot edited a comment on pull request #14765: URL: https://github.com/apache/flink/pull/14765#issuecomment-767648687 ## CI report: * 08c35397ba79eab9e80cfcb2b4d58b7df5da1f1f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12566) * 31d80dfb565af80b87945d5b717c6ea3d2b8d0ec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12568) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14760: [FLINK-20994][python] Add public method to create TableEnvironment in PyFlink.
flinkbot edited a comment on pull request #14760: URL: https://github.com/apache/flink/pull/14760#issuecomment-767543514 ## CI report: * 7ab4b72ce52f1bf451df45ca43814b221025ebbd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12559) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14727: [FLINK-19945][Connectors / FileSystem]Support sink parallelism config…
flinkbot edited a comment on pull request #14727: URL: https://github.com/apache/flink/pull/14727#issuecomment-765189922 ## CI report: * 4d9ace4ee4a4650ea186f806b0379995fee65873 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14781: [release-1.12][FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
flinkbot commented on pull request #14781: URL: https://github.com/apache/flink/pull/14781#issuecomment-768444750 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c5896f26c6106ab25c7c9c24bf9867dabca83e98 (Wed Jan 27 17:24:52 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21138) KvStateServerHandler is not invoked with user code classloader
[
https://issues.apache.org/jira/browse/FLINK-21138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273026#comment-17273026
]
Till Rohrmann commented on FLINK-21138:
---
Thanks for posting this example. I am wondering whether the problem isn't that
we are using the {{Thread.currentThread().getContextClassLoader()}}
[here|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java#L626]
instead of {{original.getClass().getClassLoader()}}. I think this should fix
the problem.
Test-wise you could take a look at the
{{PojoSerializerUpgradeTest.testPojoSerializerUpgrade}} which creates a new
class and instantiates an {{URLClassLoader}} to load it. If you then use the
test thread, it should not know about this new class.
> KvStateServerHandler is not invoked with user code classloader
> --
>
> Key: FLINK-21138
> URL: https://issues.apache.org/jira/browse/FLINK-21138
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.11.2
>Reporter: Maciej Prochniak
>Priority: Major
> Attachments: TestJob.java, stacktrace
>
>
> When using e.g. custom Kryo serializers user code classloader has to be set
> as context classloader during invocation of methods such as
> TypeSerializer.duplicat()
> KvStateServerHandler does not do this, which leads to exceptions like
> ClassNotFound etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tweise opened a new pull request #14781: [release-1.12][FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
tweise opened a new pull request #14781: URL: https://github.com/apache/flink/pull/14781 1.12 backport of https://github.com/apache/flink/pull/14711 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] raghavgautam commented on pull request #14724: [FLINK-19360] Fink startup fails when $JAVA_HOME have space
raghavgautam commented on pull request #14724: URL: https://github.com/apache/flink/pull/14724#issuecomment-768434524 @zentol I have addressed your review comment. Please take another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14777: [FLINK-21164][rest] Delete temporary jars
flinkbot edited a comment on pull request #14777: URL: https://github.com/apache/flink/pull/14777#issuecomment-768256830 ## CI report: * 8eded2f333e80f7d9ef5adaada7a3323774b8869 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.
[ https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273006#comment-17273006 ] Till Rohrmann commented on FLINK-21108: --- One way to limit the access to real servers is to not expose the Flink services to the outside world. E.g. one could use ClusterIP as the service type. Alternatively, one can activate the SSL encryption for the REST endpoint and only exchange secrets with the proxy. > Flink runtime rest server and history server webmonitor do not require > authentication. > -- > > Key: FLINK-21108 > URL: https://issues.apache.org/jira/browse/FLINK-21108 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Xiaoguang Sun >Priority: Major > Labels: pull-request-available > > Flink runtime rest server and history server webmonitor do not require > authentication. At certain scenarios, prohibiting unauthorized access is > desired. Http basic authentication can be used here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman commented on a change in pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
sjwiesman commented on a change in pull request #14780:
URL: https://github.com/apache/flink/pull/14780#discussion_r565478156
##
File path: docs/dev/project-configuration.md
##
@@ -146,6 +146,23 @@ pick the Scala version that matches their application's
Scala version.
Please refer to the [build guide]({% link flinkDev/building.md
%}#scala-versions)
for details on how to build Flink for a specific Scala version.
+Scala versions after 2.12.8 are not binary compatible with previous 2.12.x
+versions. The Flink project does therefore not offer releases for versions past
+2.12.8. If you want, you can build Flink yourself for a later Scala version.
+Refer to the [build guide]({%link flinkDev/building.md %}#scala-versions) for
+more information, as mentioned above. For this to work, you need to add
+`-Djapicmp.skip` to skip binary compatibility checks when building.
Review comment:
It was a bit wordy, I think this sounds a little clearer. I also want to
make it very clear this is out of our hands.
```suggestion
Scala versions after 2.12.8 are not binary compatible with previous 2.12.x
versions, preventing the Flink project from upgrading its 2.12.x builds
beyond 2.12.8.
Users can build Flink locally for latter Scala versions by following the
above mentioned
[build guide]({% link flinkDev/building.md %}#scala-versions). For this to
work, users
need to add `-Djapicmp.skip` to skip binary compatibility checks when
building.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tweise commented on pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
tweise commented on pull request #14711: URL: https://github.com/apache/flink/pull/14711#issuecomment-768422261 @becketqin thanks for the review. In this case `WhiteBox` is needed to check that the settings are reflected in the Kafka client objects. Should there be ideas how to solve this in a better way I will be happy to take it as follow-up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise merged pull request #14711: [FLINK-21059][kafka] KafkaSourceEnumerator does not honor consumer properties
tweise merged pull request #14711: URL: https://github.com/apache/flink/pull/14711 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
zentol commented on a change in pull request #14780:
URL: https://github.com/apache/flink/pull/14780#discussion_r565470148
##
File path: docs/flinkDev/building.md
##
@@ -132,7 +132,7 @@ Since version 1.7 Flink builds with Scala version 2.11
(default) and 2.12.
To build FLink against Scala 2.12, issue the following command:
{% highlight bash %}
-mvn clean install -DskipTests -Dscala-2.12
+mvn clean install -DskipTests -Dscala-2.12 -Dscala.version=2.12.12
Review comment:
Shouldn't we use the default scala version here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
flinkbot edited a comment on pull request #14765: URL: https://github.com/apache/flink/pull/14765#issuecomment-767648687 ## CI report: * 08c35397ba79eab9e80cfcb2b4d58b7df5da1f1f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12566) * 31d80dfb565af80b87945d5b717c6ea3d2b8d0ec UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #14767: [FLINK-21099] Introduce JobType to distinguish between batch and streaming jobs
zentol commented on a change in pull request #14767:
URL: https://github.com/apache/flink/pull/14767#discussion_r565469068
##
File path:
flink-tests/src/test/java/org/apache/flink/test/runtime/BlockingShuffleITCase.java
##
@@ -94,6 +95,7 @@ private JobGraph createJobGraph(int numRecordsToSend) {
StreamGraph streamGraph = env.getStreamGraph();
streamGraph.setGlobalDataExchangeMode(GlobalDataExchangeMode.ALL_EDGES_BLOCKING);
streamGraph.setScheduleMode(ScheduleMode.LAZY_FROM_SOURCES);
+streamGraph.setJobType(JobType.BATCH);
Review comment:
Yes, it was just a thought.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
dawidwys commented on a change in pull request #14765:
URL: https://github.com/apache/flink/pull/14765#discussion_r565462300
##
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/iterator/RocksSingleStateIterator.java
##
@@ -23,11 +23,13 @@
import javax.annotation.Nonnull;
+import java.io.Closeable;
+
/**
* Wraps a RocksDB iterator to cache it's current key and assigns an id for
the key/value state to
* the iterator. Used by {@link RocksStatesPerKeyGroupMergeIterator}.
*/
-class RocksSingleStateIterator implements AutoCloseable {
+class RocksSingleStateIterator implements Closeable {
Review comment:
got it, I think that's fine
##
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/iterator/RocksSingleStateIterator.java
##
@@ -23,11 +23,13 @@
import javax.annotation.Nonnull;
+import java.io.Closeable;
+
/**
* Wraps a RocksDB iterator to cache it's current key and assigns an id for
the key/value state to
* the iterator. Used by {@link RocksStatesPerKeyGroupMergeIterator}.
*/
-class RocksSingleStateIterator implements AutoCloseable {
+class RocksSingleStateIterator implements Closeable {
Review comment:
got it, I think that's fine to use Closeable for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-18685) JobClient.getAccumulators() blocks until streaming job has finished in local environment
[
https://issues.apache.org/jira/browse/FLINK-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272982#comment-17272982
]
Etienne Chauchot commented on FLINK-18685:
--
[~rmetzger] my pleasure ! thanks for reviewing/merging
> JobClient.getAccumulators() blocks until streaming job has finished in local
> environment
>
>
> Key: FLINK-18685
> URL: https://issues.apache.org/jira/browse/FLINK-18685
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Etienne Chauchot
>Priority: Major
> Labels: pull-request-available, starter
> Fix For: 1.13.0
>
>
> *Steps to reproduce:*
> {code:java}
> JobClient client = env.executeAsync("Test");
> CompletableFuture status = client.getJobStatus();
> LOG.info("status = " + status.get());
> CompletableFuture> accumulators =
> client.getAccumulators(StreamingJob.class.getClassLoader());
> LOG.info("accus = " + accumulators.get(5, TimeUnit.SECONDS));
> {code}
> *Actual behavior*
> The accumulators future will never complete for a streaming job when calling
> this just in your main() method from the IDE.
> *Expected behavior*
> Receive the accumulators of the running streaming job.
> The JavaDocs of the method state the following: "Accumulators can be
> requested while it is running or after it has finished.".
> While it is technically true that I can request accumulators, I was expecting
> as a user that I can access the accumulators of a running job.
> Also, I can request accumulators if I submit the job to a cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] aljoscha commented on a change in pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
aljoscha commented on a change in pull request #14765:
URL: https://github.com/apache/flink/pull/14765#discussion_r565458656
##
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksDBFullSnapshotResources.java
##
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.contrib.streaming.state.snapshot;
+
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend;
+import org.apache.flink.contrib.streaming.state.RocksIteratorWrapper;
+import
org.apache.flink.contrib.streaming.state.iterator.RocksStatesPerKeyGroupMergeIterator;
+import
org.apache.flink.contrib.streaming.state.iterator.RocksTransformingIteratorWrapper;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.runtime.state.FullSnapshotResources;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyValueStateIterator;
+import org.apache.flink.runtime.state.RegisteredKeyValueStateBackendMetaInfo;
+import org.apache.flink.runtime.state.StateSnapshotTransformer;
+import org.apache.flink.runtime.state.StreamCompressionDecorator;
+import org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot;
+import org.apache.flink.util.IOUtils;
+import org.apache.flink.util.ResourceGuard;
+
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.rocksdb.Snapshot;
+
+import javax.annotation.Nonnegative;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/** A {@link FullSnapshotResources} for the RocksDB backend. */
+class RocksDBFullSnapshotResources implements FullSnapshotResources {
+private final List stateMetaInfoSnapshots;
+private final ResourceGuard.Lease lease;
+private final Snapshot snapshot;
+private final RocksDB db;
+private final List metaData;
+
+/** Number of bytes in the key-group prefix. */
+@Nonnegative private final int keyGroupPrefixBytes;
+
+private final KeyGroupRange keyGroupRange;
+private final TypeSerializer keySerializer;
+private final StreamCompressionDecorator streamCompressionDecorator;
+
+public RocksDBFullSnapshotResources(
+ResourceGuard.Lease lease,
+Snapshot snapshot,
+List metaDataCopy,
+List stateMetaInfoSnapshots,
+RocksDB db,
+int keyGroupPrefixBytes,
+KeyGroupRange keyGroupRange,
+TypeSerializer keySerializer,
+StreamCompressionDecorator streamCompressionDecorator) {
+this.lease = lease;
+this.snapshot = snapshot;
+this.stateMetaInfoSnapshots = stateMetaInfoSnapshots;
+this.db = db;
+this.keyGroupPrefixBytes = keyGroupPrefixBytes;
+this.keyGroupRange = keyGroupRange;
+this.keySerializer = keySerializer;
+this.streamCompressionDecorator = streamCompressionDecorator;
+
+// we need to to this in the constructor, i.e. in the synchronous part
of the snapshot
+// TODO: better yet, we can do it outside the constructor
+this.metaData = fillMetaData(metaDataCopy);
+}
+
+private List fillMetaData(
+List metaDataCopy) {
+List metaData = new ArrayList<>(metaDataCopy.size());
+for (RocksDBKeyedStateBackend.RocksDbKvStateInfo rocksDbKvStateInfo :
metaDataCopy) {
+StateSnapshotTransformer stateSnapshotTransformer = null;
+if (rocksDbKvStateInfo.metaInfo instanceof
RegisteredKeyValueStateBackendMetaInfo) {
+stateSnapshotTransformer =
+((RegisteredKeyValueStateBackendMetaInfo)
rocksDbKvStateInfo.metaInfo)
+.getStateSnapshotTransformFactory()
+.createForSerializedState()
+.orElse(null);
+}
+metaData.add(new MetaData(rocksDbKvStateInfo,
stateSnapshotTransformer));
+}
[GitHub] [flink] aljoscha commented on a change in pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
aljoscha commented on a change in pull request #14765:
URL: https://github.com/apache/flink/pull/14765#discussion_r565457375
##
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/iterator/RocksStatesPerKeyGroupMergeIterator.java
##
@@ -132,10 +145,14 @@ public void next() {
final RocksIteratorWrapper rocksIterator =
rocksIteratorWithKVStateId.f0;
rocksIterator.seekToFirst();
if (rocksIterator.isValid()) {
-iteratorPriorityQueue.offer(
-new RocksSingleStateIterator(rocksIterator,
rocksIteratorWithKVStateId.f1));
+RocksSingleStateIterator wrappingIterator =
+new RocksSingleStateIterator(rocksIterator,
rocksIteratorWithKVStateId.f1);
+iteratorPriorityQueue.offer(wrappingIterator);
+closeableRegistry.registerCloseable(wrappingIterator);
+closeableRegistry.unregisterCloseable(rocksIterator);
Review comment:
See answer above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
aljoscha commented on a change in pull request #14765:
URL: https://github.com/apache/flink/pull/14765#discussion_r565455652
##
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/iterator/RocksStatesPerKeyGroupMergeIterator.java
##
@@ -106,7 +118,8 @@ public void next() {
detectNewKeyGroup(oldKey);
}
} else {
-IOUtils.closeQuietly(rocksIterator);
+IOUtils.closeQuietly(currentSubIterator);
+closeableRegistry.unregisterCloseable(currentSubIterator);
Review comment:
Thanks, I will use this pattern! And yes, it's actually important to
unregister, we have tests that verify that exactly the right amount of
`close()` calls happen on the Rocks classes. I was surprised myself. 🎊
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
aljoscha commented on a change in pull request #14765:
URL: https://github.com/apache/flink/pull/14765#discussion_r565454972
##
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/iterator/RocksSingleStateIterator.java
##
@@ -23,11 +23,13 @@
import javax.annotation.Nonnull;
+import java.io.Closeable;
+
/**
* Wraps a RocksDB iterator to cache it's current key and assigns an id for
the key/value state to
* the iterator. Used by {@link RocksStatesPerKeyGroupMergeIterator}.
*/
-class RocksSingleStateIterator implements AutoCloseable {
+class RocksSingleStateIterator implements Closeable {
Review comment:
I made it `Closeable` because the `RocksStatesPerKeyGroupMergeIterator`
now uses a `CloseableRegistry` which requires things to be `Closeable`.
Alternatively, I could change `CloseableRegistry` to take `AutoCloseables`
but I felt that would be the bigger change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
aljoscha commented on a change in pull request #14765:
URL: https://github.com/apache/flink/pull/14765#discussion_r565454247
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyValueStateIterator.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+/**
+ * Iterator that over all key-value state entries in a {@link
KeyedStateBackend}. For use during
+ * snapshotting.
+ *
+ * This is required to partition all states into contiguous key-groups. The
resulting iteration
+ * sequence is ordered by (key-group, kv-state).
+ */
+public interface KeyValueStateIterator extends AutoCloseable {
+
+/**
+ * Advances the iterator. Should only be called if {@link #isValid()}
returned true. Valid flag
+ * can only change after calling {@link #next()}.
+ */
+void next();
+
+/** Returns the key-group for the current key. */
+int keyGroup();
+
+byte[] key();
+
+byte[] value();
+
+/** Returns the Id of the K/V state to which the current key belongs. */
+int kvStateId();
+
+/**
+ * Indicates if current key starts a new k/v-state, i.e. belong to a
different k/v-state than
+ * it's predecessor.
+ *
+ * @return true iff the current key belong to a different k/v-state than
it's predecessor.
+ */
+boolean isNewKeyValueState();
+
+/**
+ * Indicates if current key starts a new key-group, i.e. belong to a
different key-group than
+ * it's predecessor.
+ *
+ * @return true iff the current key belong to a different key-group than
it's predecessor.
+ */
+boolean isNewKeyGroup();
+
+/**
+ * Check if the iterator is still valid. Getters like {@link #key()},
{@link #value()}, etc. as
+ * well as {@link #next()} should only be called if valid returned true.
Should be checked after
+ * each call to {@link #next()} before accessing iterator state.
+ *
+ * @return True iff this iterator is valid.
+ */
+boolean isValid();
+
+@Override
+void close();
Review comment:
I kept this from `RocksStatesPerKeyGroupMergeIterator`, which also
overrode `close()` without exception. And some of the code that uses it expects
it not to throw.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
flinkbot edited a comment on pull request #14765: URL: https://github.com/apache/flink/pull/14765#issuecomment-767648687 ## CI report: * 08c35397ba79eab9e80cfcb2b4d58b7df5da1f1f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12566) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on a change in pull request #14737:
URL: https://github.com/apache/flink/pull/14737#discussion_r565363984
##
File path:
flink-end-to-end-tests/flink-glue-schema-registry-test/src/main/java/org.apache.flink.glue.schema.registry.test/GSRKinesisPubsubClient.java
##
@@ -0,0 +1,185 @@
+/*
Review comment:
It seems that the package name is not properly encoded into
subdirectories.
Part of the directory name of this file is
`org.apache.flink.glue.schema.registry.test`, but it should be
`org/apache/flink/glue/schema/registry/test`. This might be difficult to see in
some IDEs, as they are replacing this directory structure with the dot-notation.
##
File path:
flink-end-to-end-tests/flink-glue-schema-registry-test/src/main/java/org.apache.flink.glue.schema.registry.test/GlueSchemaRegistryExampleTest.java
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.glue.schema.registry.test;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.java.utils.ParameterTool;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.junit.Assert;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
+
+/** Test driver for {@link GlueSchemaRegistryExample#main}. */
+public class GlueSchemaRegistryExampleTest {
+private static final Logger LOG =
LoggerFactory.getLogger(GlueSchemaRegistryExampleTest.class);
+
+public static void main(String[] args) throws Exception {
+LOG.info("System properties: {}", System.getProperties());
+final ParameterTool parameterTool = ParameterTool.fromArgs(args);
+
+String inputStream = parameterTool.getRequired("input-stream");
+String outputStream = parameterTool.getRequired("output-stream");
+
+GSRKinesisPubsubClient pubsub = new
GSRKinesisPubsubClient(parameterTool.getProperties());
+pubsub.createTopic(inputStream, 2, parameterTool.getProperties());
+pubsub.createTopic(outputStream, 2, parameterTool.getProperties());
+
+// The example job needs to start after streams are created and run in
parallel to the
+// validation logic.
+// The thread that runs the job won't terminate, we don't have a job
reference to cancel it.
+// Once results are validated, the driver main thread will exit;
job/cluster will be
+// terminated from script.
+final AtomicReference executeException = new
AtomicReference<>();
+Thread executeThread =
+new Thread(
+() -> {
+try {
+GlueSchemaRegistryExample.main(args);
+// this message won't appear in the log,
+// job is terminated when shutting down cluster
+LOG.info("executed program");
+} catch (Exception e) {
+executeException.set(e);
+}
+});
+executeThread.start();
+
+List messages = getRecords();
+for (GenericRecord msg : messages) {
+
pubsub.sendMessage(GlueSchemaRegistryExample.getSchema().toString(),
inputStream, msg);
+}
+LOG.info("generated records");
+
+Deadline deadline = Deadline.fromNow(Duration.ofSeconds(60));
+List results = pubsub.readAllMessages(outputStream);
+while (deadline.hasTimeLeft()
+&& executeException.get() == null
+&& results.size() < messages.size()) {
+LOG.info("waiting for results..");
+Thread.sleep(1000);
+results = pubsub.readAllMessages(outputStream);
+}
+
+if (executeException.get() != null) {
+throw executeException.get();
+}
+
+LOG.info("results: {}", results);
+Assert.assertEquals(
+
[GitHub] [flink] aljoscha commented on a change in pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
aljoscha commented on a change in pull request #14765:
URL: https://github.com/apache/flink/pull/14765#discussion_r565449964
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/FullSnapshotUtil.java
##
@@ -16,13 +16,13 @@
* limitations under the License.
*/
-package org.apache.flink.contrib.streaming.state.snapshot;
+package org.apache.flink.runtime.state;
/**
- * Utility methods and constants around RocksDB creating and restoring
snapshots for {@link
- * org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend}.
+ * Utility methods and constants around creating and restoring full snapshots
using {@link
+ * FullSnapshotAsyncWriter}.
*/
-public class RocksSnapshotUtil {
+public class FullSnapshotUtil {
/** File suffix of sstable files. */
public static final String SST_FILE_SUFFIX = ".sst";
Review comment:
True, I will have to move some of them back to a Rocks-specific
`RocksSnapshotUtil`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14775: [FLINK-20964][python] Introduce PythonStreamGroupWindowAggregateOperator
flinkbot edited a comment on pull request #14775: URL: https://github.com/apache/flink/pull/14775#issuecomment-768234660 ## CI report: * 7df0cb198efffa5c5dd7d6cbe98e4db63f8e4269 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12554) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14765: [FLINK-21151] Extract common full-snapshot writer from RocksDB full-snapshot strategy
flinkbot edited a comment on pull request #14765: URL: https://github.com/apache/flink/pull/14765#issuecomment-767648687 ## CI report: * 52588009ab10795ee7c7e576ad11879760350871 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12523) * 08c35397ba79eab9e80cfcb2b4d58b7df5da1f1f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.
[ https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272950#comment-17272950 ] Xiaoguang Sun edited comment on FLINK-21108 at 1/27/21, 3:53 PM: - At this time, anybody who knows the endpoint of a running Flink cluster can access it. This makes it easier to be compromised when there is vulnerability, CVE-2020-17518 for example. In addition, people who has access to web console can terminate or even submit new job which essentially makes it possible to run arbitrary code on production environment. In addition, people can introspect configuration of running jobs which might contain sensitive information as well. For these reasons, it is kind of important to restrict Flink console to be only available to authenticated users. Reverse proxy is one way of doing it, but we probably need to figure out a way to completely hide the real server behind such authentication proxy so malicious users can't by pass proxy and access Flink console directly. This problem is getting worse when users are running Flink on Kubernetes, the highly dynamic nature of Kubernetes will make hardening Flink console a tough job because the address of Kubernetes ingress server might change as well. If restricting access to Flink console is something we have to do, we will have to deal with this scenario eventually. was (Author: xexplorer): At this time, anybody who knows the endpoint of a running Flink cluster can access it. This makes it easier to be compromised when there is vulnerability, CVE-2020-17518 for example. In addition, people who has access to web console can terminate or even submit new job which essentially make it possible to run arbitrary code on production environment. In addition, people can introspect configuration of running jobs which might contain sensitive information as well. For these reasons, it is kind of important to restrict Flink console to be only available to authenticated users. Reverse proxy is one way of doing it, but we probably need to figure out a way to completely hide the real server behind such authentication proxy so malicious users can't by pass proxy and access Flink console directly. This problem is getting worse when users are running Flink on Kubernetes, the highly dynamic nature of Kubernetes will make hardening Flink console a tough job because the address of Kubernetes ingress server might change as well. If restricting access to Flink console is something we have to do, we will have to deal with this scenario eventually. > Flink runtime rest server and history server webmonitor do not require > authentication. > -- > > Key: FLINK-21108 > URL: https://issues.apache.org/jira/browse/FLINK-21108 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Xiaoguang Sun >Priority: Major > Labels: pull-request-available > > Flink runtime rest server and history server webmonitor do not require > authentication. At certain scenarios, prohibiting unauthorized access is > desired. Http basic authentication can be used here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.
[ https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272950#comment-17272950 ] Xiaoguang Sun edited comment on FLINK-21108 at 1/27/21, 3:52 PM: - At this time, anybody who knows the endpoint of a running Flink cluster can access it. This makes it easier to be compromised when there is vulnerability, CVE-2020-17518 for example. In addition, people who has access to web console can terminate or even submit new job which essentially make it possible to run arbitrary code on production environment. In addition, people can introspect configuration of running jobs which might contain sensitive information as well. For these reasons, it is kind of important to restrict Flink console to be only available to authenticated users. Reverse proxy is one way of doing it, but we probably need to figure out a way to completely hide the real server behind such authentication proxy so malicious users can't by pass proxy and access Flink console directly. This problem is getting worse when users are running Flink on Kubernetes, the highly dynamic nature of Kubernetes will make hardening Flink console a tough job because the address of Kubernetes ingress server might change as well. If restricting access to Flink console is something we have to do, we will have to deal with this scenario eventually. was (Author: xexplorer): At this time, anybody who knows the endpoint of a running Flink cluster can access it. This makes it easier to be compromised when there is vulnerability, CVE-2020-17518 for example. In addition, people how has access to web console can terminate or even submit new job which essentially make it possible to run arbitrary code on production environment. In addition, people can introspect configuration of running jobs which might contain sensitive information as well. For these reasons, it is kind of important to restrict Flink console to be only available to authenticated users. Reverse proxy is one way of doing it, but we probably need to figure out a way to completely hide the real server behind such authentication proxy so malicious users can't by pass proxy and access Flink console directly. This problem is getting worse when users are running Flink on Kubernetes, the highly dynamic nature of Kubernetes will make hardening Flink console a tough job because the address of Kubernetes ingress server might change as well. If restricting access to Flink console is something we have to do, we will have to deal with this scenario eventually. > Flink runtime rest server and history server webmonitor do not require > authentication. > -- > > Key: FLINK-21108 > URL: https://issues.apache.org/jira/browse/FLINK-21108 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Xiaoguang Sun >Priority: Major > Labels: pull-request-available > > Flink runtime rest server and history server webmonitor do not require > authentication. At certain scenarios, prohibiting unauthorized access is > desired. Http basic authentication can be used here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.
[ https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272950#comment-17272950 ] Xiaoguang Sun commented on FLINK-21108: --- At this time, anybody who knows the endpoint of a running Flink cluster can access it. This makes it easier to be compromised when there is vulnerability, CVE-2020-17518 for example. In addition, people how has access to web console can terminate or even submit new job which essentially make it possible to run arbitrary code on production environment. In addition, people can introspect configuration of running jobs which might contain sensitive information as well. For these reasons, it is kind of important to restrict Flink console to be only available to authenticated users. Reverse proxy is one way of doing it, but we probably need to figure out a way to completely hide the real server behind such authentication proxy so malicious users can't by pass proxy and access Flink console directly. This problem is getting worse when users are running Flink on Kubernetes, the highly dynamic nature of Kubernetes will make hardening Flink console a tough job because the address of Kubernetes ingress server might change as well. If restricting access to Flink console is something we have to do, we will have to deal with this scenario eventually. > Flink runtime rest server and history server webmonitor do not require > authentication. > -- > > Key: FLINK-21108 > URL: https://issues.apache.org/jira/browse/FLINK-21108 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Xiaoguang Sun >Priority: Major > Labels: pull-request-available > > Flink runtime rest server and history server webmonitor do not require > authentication. At certain scenarios, prohibiting unauthorized access is > desired. Http basic authentication can be used here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14767: [FLINK-21099] Introduce JobType to distinguish between batch and streaming jobs
flinkbot edited a comment on pull request #14767: URL: https://github.com/apache/flink/pull/14767#issuecomment-767740322 ## CI report: * 12db140bd9ec875810a27e88b840f2969d34fb63 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12562) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * 3885663038156d52aec21062a9743009fb875cc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12535) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-21006) HBaseTablePlanTest tests failed in haoop 3.1.3 with "java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;
[
https://issues.apache.org/jira/browse/FLINK-21006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger closed FLINK-21006.
--
Resolution: Fixed
Resolved in
https://github.com/apache/flink/commit/dac3e72c6777678af8d2dbbbd61aaff1e385981d
> HBaseTablePlanTest tests failed in haoop 3.1.3 with
> "java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V"
> -
>
> Key: FLINK-21006
> URL: https://issues.apache.org/jira/browse/FLINK-21006
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.13.0
>Reporter: Huang Xingbo
>Assignee: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12159&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=bfbc6239-57a0-5db0-63f3-41551b4f7d51]
> {code:java}
> 2021-01-15T22:48:58.1843544Z Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
> 2021-01-15T22:48:58.1844358Z at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
> 2021-01-15T22:48:58.1845035Z at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
> 2021-01-15T22:48:58.1845805Z at
> org.apache.flink.connector.hbase.options.HBaseOptions.getHBaseConfiguration(HBaseOptions.java:157)
> 2021-01-15T22:48:58.1846960Z at
> org.apache.flink.connector.hbase1.HBase1DynamicTableFactory.createDynamicTableSource(HBase1DynamicTableFactory.java:73)
> 2021-01-15T22:48:58.1848020Z at
> org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:119)
> 2021-01-15T22:48:58.1848574Z ... 49 more
> {code}
> The exception seems that the different version of guava caused.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger closed pull request #14770: [FLINK-21006] Fix hbase 1.4 tests on Hadoop 3.x
rmetzger closed pull request #14770: URL: https://github.com/apache/flink/pull/14770 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * 3885663038156d52aec21062a9743009fb875cc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12535) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18685) JobClient.getAccumulators() blocks until streaming job has finished in local environment
[
https://issues.apache.org/jira/browse/FLINK-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger closed FLINK-18685.
--
Fix Version/s: 1.13.0
Resolution: Fixed
> JobClient.getAccumulators() blocks until streaming job has finished in local
> environment
>
>
> Key: FLINK-18685
> URL: https://issues.apache.org/jira/browse/FLINK-18685
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Etienne Chauchot
>Priority: Major
> Labels: pull-request-available, starter
> Fix For: 1.13.0
>
>
> *Steps to reproduce:*
> {code:java}
> JobClient client = env.executeAsync("Test");
> CompletableFuture status = client.getJobStatus();
> LOG.info("status = " + status.get());
> CompletableFuture> accumulators =
> client.getAccumulators(StreamingJob.class.getClassLoader());
> LOG.info("accus = " + accumulators.get(5, TimeUnit.SECONDS));
> {code}
> *Actual behavior*
> The accumulators future will never complete for a streaming job when calling
> this just in your main() method from the IDE.
> *Expected behavior*
> Receive the accumulators of the running streaming job.
> The JavaDocs of the method state the following: "Accumulators can be
> requested while it is running or after it has finished.".
> While it is technically true that I can request accumulators, I was expecting
> as a user that I can access the accumulators of a running job.
> Also, I can request accumulators if I submit the job to a cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18685) JobClient.getAccumulators() blocks until streaming job has finished in local environment
[
https://issues.apache.org/jira/browse/FLINK-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272922#comment-17272922
]
Robert Metzger commented on FLINK-18685:
Resolved in
https://github.com/apache/flink/commit/beae8e7a20dfa915ce02fd3e5186d4b5e71fbbef.
Thanks a lot for the fix Etienne!
> JobClient.getAccumulators() blocks until streaming job has finished in local
> environment
>
>
> Key: FLINK-18685
> URL: https://issues.apache.org/jira/browse/FLINK-18685
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Etienne Chauchot
>Priority: Major
> Labels: pull-request-available, starter
>
> *Steps to reproduce:*
> {code:java}
> JobClient client = env.executeAsync("Test");
> CompletableFuture status = client.getJobStatus();
> LOG.info("status = " + status.get());
> CompletableFuture> accumulators =
> client.getAccumulators(StreamingJob.class.getClassLoader());
> LOG.info("accus = " + accumulators.get(5, TimeUnit.SECONDS));
> {code}
> *Actual behavior*
> The accumulators future will never complete for a streaming job when calling
> this just in your main() method from the IDE.
> *Expected behavior*
> Receive the accumulators of the running streaming job.
> The JavaDocs of the method state the following: "Accumulators can be
> requested while it is running or after it has finished.".
> While it is technically true that I can request accumulators, I was expecting
> as a user that I can access the accumulators of a running job.
> Also, I can request accumulators if I submit the job to a cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272921#comment-17272921
]
Aljoscha Krettek commented on FLINK-21132:
--
It seems to me we just need to make sure that {{endOfInput()}} is not called
for the stop-with-savepoint case. That method must only be called when it is
*truly* the end of input, that is when the source and all the predecessor
operators have shut down. What do you think? With this, the Jira Issue should
be easy enough to fix.
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Priority: Major
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger closed pull request #14558: [FLINK-18685] [runtime] Make MiniClusterJobClient#getAccumulators non-blocking in Streaming mode
rmetzger closed pull request #14558: URL: https://github.com/apache/flink/pull/14558 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
sjwiesman commented on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-768343868 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
flinkbot edited a comment on pull request #14780: URL: https://github.com/apache/flink/pull/14780#issuecomment-768329420 ## CI report: * 8251a890daad47333167e3998fc697ffd45eca8f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12563) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14767: [FLINK-21099] Introduce JobType to distinguish between batch and streaming jobs
flinkbot edited a comment on pull request #14767: URL: https://github.com/apache/flink/pull/14767#issuecomment-767740322 ## CI report: * 7b5d7b37c9a10af67665068053bed59ecb2fccc8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12527) * 12db140bd9ec875810a27e88b840f2969d34fb63 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12562) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-768331002 Thanks a lot for the clarification This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20701) print connector supports outputting seriailzed record
[ https://issues.apache.org/jira/browse/FLINK-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-20701. --- Resolution: Invalid Thanks [~zoucao], I will close this issue then. > print connector supports outputting seriailzed record > - > > Key: FLINK-20701 > URL: https://issues.apache.org/jira/browse/FLINK-20701 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Affects Versions: 1.13.0 >Reporter: zoucao >Priority: Major > > Hi all， print connector is designed for debugging，and it gives us better > access to data. If I want to debug source connector and SQL statement, I will > use it to print the result. However, I don't have a good choice to observe > the seriailzed record, which will be sent to external connector when > debugging. Consequently, we 'd better make print connetcor support outputting > seriailzed record. We can use format=XXX to identify a seriailzed schema, > help user to debug and understand the process of serialization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[ https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272898#comment-17272898 ] Jark Wu commented on FLINK-21045: - Thanks for your reply [~twalthr]. I don't have strong opinion on CREATE/DROP MODULE vs. LOAD/UNLOAD MODULE. I would like to hear more thoughs from others, cc [~lirui] [~lzljs3620320] > Support 'load module' and 'unload module' SQL syntax > > > Key: FLINK-21045 > URL: https://issues.apache.org/jira/browse/FLINK-21045 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.13.0 >Reporter: Nicholas Jiang >Assignee: Jane Chan >Priority: Major > Fix For: 1.13.0 > > > At present, Flink SQL doesn't support the 'load module' and 'unload module' > SQL syntax. It's necessary for uses in the situation that users load and > unload user-defined module through table api or sql client. > SQL syntax has been proposed in FLIP-68: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
flinkbot commented on pull request #14780: URL: https://github.com/apache/flink/pull/14780#issuecomment-768329420 ## CI report: * 8251a890daad47333167e3998fc697ffd45eca8f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14767: [FLINK-21099] Introduce JobType to distinguish between batch and streaming jobs
flinkbot edited a comment on pull request #14767: URL: https://github.com/apache/flink/pull/14767#issuecomment-767740322 ## CI report: * 7b5d7b37c9a10af67665068053bed59ecb2fccc8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12527) * 12db140bd9ec875810a27e88b840f2969d34fb63 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20951) IllegalArgumentException when reading Hive parquet table if condition not contain all partitioned fields
[
https://issues.apache.org/jira/browse/FLINK-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272893#comment-17272893
]
Jude Zhu commented on FLINK-20951:
--
I' alread set
*tableEnv.getConfig().getConfiguration().setString("table.exec.hive.fallback-mapred-reader",
"true");*
*when i only append one partition condition。all of two* *partition***
the exception is：
21/01/27 20:53:26 ERROR base.source.reader.fetcher.SplitFetcherManager:
Received uncaught exception.21/01/27 20:53:26 ERROR
base.source.reader.fetcher.SplitFetcherManager: Received uncaught
exception.java.lang.RuntimeException: SplitFetcher thread 79 received
unexpected exception while polling the records at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:146)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101)
[TalosEngine-Prod-1.12-2.1.11.jar:?] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_121] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_121] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]Caused by:
java.io.IOException: Filesystem closed at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:858)
~[hadoop-hdfs-2.6.0-cdh5.11.0.jar:?] at
org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:707)
~[hadoop-hdfs-2.6.0-cdh5.11.0.jar:?] at
java.io.FilterInputStream.close(FilterInputStream.java:181) ~[?:1.8.0_121] at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:432)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:371)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:252)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:99)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:85)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connectors.hive.read.HiveMapredSplitReader.(HiveMapredSplitReader.java:113)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connectors.hive.read.HiveBulkFormatAdapter$HiveReader.(HiveBulkFormatAdapter.java:309)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connectors.hive.read.HiveBulkFormatAdapter$HiveReader.(HiveBulkFormatAdapter.java:288)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connectors.hive.read.HiveBulkFormatAdapter$HiveMapRedBulkFormat.createReader(HiveBulkFormatAdapter.java:265)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connectors.hive.read.HiveBulkFormatAdapter$HiveMapRedBulkFormat.createReader(HiveBulkFormatAdapter.java:258)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connectors.hive.read.HiveBulkFormatAdapter.createReader(HiveBulkFormatAdapter.java:108)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connectors.hive.read.HiveBulkFormatAdapter.createReader(HiveBulkFormatAdapter.java:63)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connector.file.src.impl.FileSourceSplitReader.checkSplitOrStartNext(FileSourceSplitReader.java:112)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:65)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:56)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:138)
~[TalosEngine-Prod-1.12-2.1.11.jar:?] ... 6 more
and
21/01/27 20:53:26 ERROR runtime.io.network.partition.ResultPartition: Error
during release of result subpartition:
/data1/yarn/local/usercache/venus/appcache/application_1605868815011_5695885/flink-netty-shuffle-a2999438-b901-472e-a19a-99d025c13c20/0ae8a74acd295d92c30242edc432e358.channel21/01/27
20:53:26 ERROR runtime.io.network.partition.ResultPartition: Error during
release of result subpartition:
/data1/yarn/local/usercache/venus/appcache/application_1605868815011_5695885/flink-netty-shuffle-a2999438-b901-472e-a19a-99d025c13c20/0ae8a74acd295d92c30242edc432e358.ch
[GitHub] [flink] flinkbot commented on pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
flinkbot commented on pull request #14780: URL: https://github.com/apache/flink/pull/14780#issuecomment-768319433 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8251a890daad47333167e3998fc697ffd45eca8f (Wed Jan 27 14:24:16 UTC 2021) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-12461).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] aljoscha opened a new pull request #14780: [FLINK-12461] Document Scala 2.12.x situation
aljoscha opened a new pull request #14780: URL: https://github.com/apache/flink/pull/14780 ## What is the purpose of the change Scala versions after 2.12.8 are not binary compatible with previous versions, we link to the relevant release notes and point users towards building their own release using a more recent version. It's a pure documentation change, no testing needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
