[jira] [Commented] (FLINK-19925) Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[
https://issues.apache.org/jira/browse/FLINK-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227154#comment-17227154
]
Yingjie Cao commented on FLINK-19925:
-
Usually, it indicates an unstable network which may be caused by high cpu usage
or high network pressure. As discussed offline, we will do some further test to
investigate it.
> Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
> --
>
> Key: FLINK-19925
> URL: https://issues.apache.org/jira/browse/FLINK-19925
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.12.0
>Reporter: godfrey he
>Priority: Major
>
> Errors$NativeIoException will occur sometime when we run TPCDS based on
> master, the full exception stack is
> {code:java}
> Caused by:
> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
> readAddress(..) failed: Connection reset by peer (connection to 'xxx')
> at
> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.exceptionCaught(CreditBasedPartitionRequestClientHandler.java:173)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:818)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102]
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors$NativeIoException:
> readAddress(..) failed: Connection reset by peer
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20013) BoundedBlockingSubpartition may leak network buffer if task is failed or canceled
Yingjie Cao created FLINK-20013: --- Summary: BoundedBlockingSubpartition may leak network buffer if task is failed or canceled Key: FLINK-20013 URL: https://issues.apache.org/jira/browse/FLINK-20013 Project: Flink Issue Type: Bug Components: Runtime / Network Reporter: Yingjie Cao Fix For: 1.12.0 BoundedBlockingSubpartition may leak network buffer if task is failed or canceled. We need to recycle the current BufferConsumer when task is failed or canceled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13841: [FLINK-19850] Add e2e tests for the new File Sink in the streaming mode
flinkbot edited a comment on pull request #13841: URL: https://github.com/apache/flink/pull/13841#issuecomment-718698511 ## CI report: * ba30909849312f20c9ae3cdf2f3b270195eeaac9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9130) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13935: [FLINK-18926][python][doc] Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section.
flinkbot edited a comment on pull request #13935: URL: https://github.com/apache/flink/pull/13935#issuecomment-722244949 ## CI report: * 3758f3ad3644ca64bbb7e7bb40db4db903354236 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9064) * 65c1e98a7e73ef54b0ade9109ea567e8aa4bb180 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9131) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13955: [FLINK-19934][Connector] Add a new API: SplitEnumeratorContext.runInC…
flinkbot commented on pull request #13955: URL: https://github.com/apache/flink/pull/13955#issuecomment-722804935 ## CI report: * a4a7d6774aa62d7f3f7fbd850f3024ec7cf5a78d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20014) Resolve CVE-2020-11022 and CVE-2020-11023 in scala-compiler
Alan Leong created FLINK-20014: -- Summary: Resolve CVE-2020-11022 and CVE-2020-11023 in scala-compiler Key: FLINK-20014 URL: https://issues.apache.org/jira/browse/FLINK-20014 Project: Flink Issue Type: Improvement Components: Build System Reporter: Alan Leong Update scala version to resolve CVE-2020-11022 and CVE-2020-11023 in scala-compiler. This issue was addressed in Scala 2.12.12 [https://github.com/scala/scala/pull/8963.|https://github.com/scala/bug/issues/11974] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13955: [FLINK-19934][Connector] Add a new API: SplitEnumeratorContext.runInC…
flinkbot edited a comment on pull request #13955: URL: https://github.com/apache/flink/pull/13955#issuecomment-722804935 ## CI report: * a4a7d6774aa62d7f3f7fbd850f3024ec7cf5a78d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9132) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19940) Task names on web UI should change when an operator chain is chained with sources
[ https://issues.apache.org/jira/browse/FLINK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caizhi Weng updated FLINK-19940: Issue Type: Bug (was: Improvement) > Task names on web UI should change when an operator chain is chained with > sources > - > > Key: FLINK-19940 > URL: https://issues.apache.org/jira/browse/FLINK-19940 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Reporter: Caizhi Weng >Priority: Major > Fix For: 1.12.0 > > > Currently task names on web UI doesn't change even if the operator chain is > chained with sources. We should change its name to show that the sources are > chained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13449: [FLINK-19282][table sql/planner]Supports watermark push down with Wat…
flinkbot edited a comment on pull request #13449: URL: https://github.com/apache/flink/pull/13449#issuecomment-696509354 ## CI report: * 614d1934dd815569b08e7ed0546c0fee1ebcfd18 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9074) * 15f99c140a47c9d9e41504f3f417ce5d63ecc5af UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] TsReaper removed a comment on pull request #13942: [FLINK-19990][table-planner-blink] MultipleInputNodeCreationProcessor#isChainableSource now considers DataStreamScanProvider
TsReaper removed a comment on pull request #13942: URL: https://github.com/apache/flink/pull/13942#issuecomment-722334386 Thanks for the review @JingsongLi , please do not squash the commits when merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Shawn-Hx commented on pull request #13949: [hotfix][doc] Chage link format in some docs
Shawn-Hx commented on pull request #13949:
URL: https://github.com/apache/flink/pull/13949#issuecomment-722825770
Hi, @sjwiesman
I change the link format from `{{ site.baseurl }}/.../xxx.html` to `{% link
.../xxx.md %}` in some doc directories.
The `build_docs.sh` can be executed without errors.
Please take a look.
Thanks~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #13864: [FLINK-19543][k8s] Implement RunningJobsRegistry, JobGraphStore based on Kubernetes API
wangyang0918 commented on pull request #13864: URL: https://github.com/apache/flink/pull/13864#issuecomment-722834851 @tillrohrmann @xintongsong Thanks for your fruitful comments. I have addressed all of them and force pushed. Please have a check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13449: [FLINK-19282][table sql/planner]Supports watermark push down with Wat…
flinkbot edited a comment on pull request #13449: URL: https://github.com/apache/flink/pull/13449#issuecomment-696509354 ## CI report: * 614d1934dd815569b08e7ed0546c0fee1ebcfd18 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9074) * 15f99c140a47c9d9e41504f3f417ce5d63ecc5af Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9134) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13942: [FLINK-19990][table-planner-blink] MultipleInputNodeCreationProcessor#isChainableSource now considers DataStreamScanProvider
flinkbot edited a comment on pull request #13942: URL: https://github.com/apache/flink/pull/13942#issuecomment-722358038 ## CI report: * 30b3bcd2da00b89794fe83bee5c5c0113949bbfa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9110) * bdf73f2711715bf90e19bd8387ee86e15216dabb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii opened a new pull request #13956: [FLINK-20006] Fix the unstable FileSinkITCase
gaoyunhaii opened a new pull request #13956: URL: https://github.com/apache/flink/pull/13956 ## What is the purpose of the change This PR fixes the unstable `FileSinkITCase`. The original implementation has some logical error, e.g., it does not keep the state for the source function, and for streaming mode it does not ensure some data are committed in the first attempt. This cause the sink either does not test the actual scenario or cause repeat commits. ## Brief change log - 1de2eca68fe489aadb55ed3084d18f4cd7ea723f fixes the unstable ITCase. ## Verifying this change *(Please pick either of the following options)* This change is a test fix. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive):**no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20006) FileSinkITCase.testFileSink: The record 0 should occur 4 times, but only occurs 8time expected:<4> but was:<8>
[
https://issues.apache.org/jira/browse/FLINK-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20006:
---
Labels: pull-request-available test-stability (was: test-stability)
> FileSinkITCase.testFileSink: The record 0 should occur 4 times, but only
> occurs 8time expected:<4> but was:<8>
> ---
>
> Key: FLINK-20006
> URL: https://issues.apache.org/jira/browse/FLINK-20006
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9082&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=62110053-334f-5295-a0ab-80dd7e2babbf
> {code}
> 2020-11-05T13:31:16.7006473Z [ERROR] Tests run: 4, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 5.565 s <<< FAILURE! - in
> org.apache.flink.connector.file.sink.FileSinkITCase
> 2020-11-05T13:31:16.7007237Z [ERROR] testFileSink[executionMode = STREAMING,
> triggerFailover = true](org.apache.flink.connector.file.sink.FileSinkITCase)
> Time elapsed: 0.548 s <<< FAILURE!
> 2020-11-05T13:31:16.7007897Z java.lang.AssertionError: The record 0 should
> occur 4 times, but only occurs 8time expected:<4> but was:<8>
> 2020-11-05T13:31:16.7008317Z at org.junit.Assert.fail(Assert.java:88)
> 2020-11-05T13:31:16.7008644Z at
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2020-11-05T13:31:16.7008987Z at
> org.junit.Assert.assertEquals(Assert.java:645)
> 2020-11-05T13:31:16.7009392Z at
> org.apache.flink.connector.file.sink.FileSinkITCase.checkResult(FileSinkITCase.java:218)
> 2020-11-05T13:31:16.7009889Z at
> org.apache.flink.connector.file.sink.FileSinkITCase.testFileSink(FileSinkITCase.java:132)
> 2020-11-05T13:31:16.7010316Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13956: [FLINK-20006] Fix the unstable FileSinkITCase
flinkbot commented on pull request #13956: URL: https://github.com/apache/flink/pull/13956#issuecomment-722847196 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1de2eca68fe489aadb55ed3084d18f4cd7ea723f (Fri Nov 06 05:36:48 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20006).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518533440
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##
@@ -30,9 +34,28 @@
*/
public final class TestingRetrievableStateStorageHelper implements RetrievableStateStorageHelper {
+ private FunctionWithException retrieveStateFunction;
+
+ private RunnableWithException discardStateRunnable;
+
+ private Function getStateSizeFunction;
Review comment:
Yes, we may need to change these functions. For example, we want to
throw an exception when `retrieveState`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13864: [FLINK-19543][k8s] Implement RunningJobsRegistry, JobGraphStore based on Kubernetes API
xintongsong commented on a change in pull request #13864:
URL: https://github.com/apache/flink/pull/13864#discussion_r518531957
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/persistence/StateHandleStore.java
##
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.persistence;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.RetrievableStateHandle;
+import org.apache.flink.util.FlinkException;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Class which stores state via the provided {@link
RetrievableStateStorageHelper} and writes the
+ * returned state handle to distributed coordination system(e.g. Zookeeper,
Kubernetes, etc.).
+ *
+ * To avoid concurrent modification issues, the implementation needs to
ensure that only the leader could
+ * update the state store.
+ *
+ * Even some methods name contains the "lock"(e.g. {@link #getAndLock}), it
does not mean the implementation has to
+ * actually lock a specific state handle. Also it could have an empty
implementation for release operation.
+ *
+ * @param Type of state
+ * @param Type of {@link ResourceVersion}
+ */
+public interface StateHandleStore {
Review comment:
nit:
```
R extends ResourceVersion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dianfu closed pull request #13945: [FLINK-19284][python][doc] Add documentation about how to use Python UDF in the Java Table API.
dianfu closed pull request #13945: URL: https://github.com/apache/flink/pull/13945 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19284) Add documentation about how to use Python UDF in the Java Table API
[ https://issues.apache.org/jira/browse/FLINK-19284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-19284. --- Fix Version/s: (was: 1.11.3) Resolution: Fixed Merged to master via 51cd11e70205eea1e4f88041d6edfbf2cef225e2 > Add documentation about how to use Python UDF in the Java Table API > --- > > Key: FLINK-19284 > URL: https://issues.apache.org/jira/browse/FLINK-19284 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Documentation >Reporter: Dian Fu >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13864: [FLINK-19543][k8s] Implement RunningJobsRegistry, JobGraphStore based on Kubernetes API
flinkbot edited a comment on pull request #13864: URL: https://github.com/apache/flink/pull/13864#issuecomment-719924065 ## CI report: * f5ee007a4e17a98cc6252e0f14c023d657b9438a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9089) * 87a51133e4aaa9b4a6867686868d5467b91add6f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518538862
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointIDCounter.java
##
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.highavailability;
+
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMap;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesException;
+import
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector;
+import org.apache.flink.runtime.checkpoint.CheckpointIDCounter;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static
org.apache.flink.kubernetes.utils.Constants.CHECKPOINT_COUNTER_KEY;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * {@link CheckpointIDCounter} implementation for Kubernetes. The counter will
be stored in
+ * JobManager-{@link org.apache.flink.api.common.JobID}-leader ConfigMap. The
key is
+ * {@link org.apache.flink.kubernetes.utils.Constants#CHECKPOINT_COUNTER_KEY},
+ * and value is counter value.
+ */
+public class KubernetesCheckpointIDCounter implements CheckpointIDCounter {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(KubernetesCheckpointIDCounter.class);
+
+ private final FlinkKubeClient kubeClient;
+
+ private final String configMapName;
+
+ private final String lockIdentity;
+
+ private volatile boolean running;
Review comment:
Yes. You are right. Just like `DefaultCompletedCheckpointStore`,
`KubernetesCheckpointIDCounter` is also not thread-safe. So `volatile` of
`running` is unnecessary.
BTW, I think `CheckpointCoordinator` could guarantee that there's no
concurrent `shutdown` and `getAndIncrement`. I will simply remove the
`volatile`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] vintageplayer commented on pull request #13951: [FLINK-19998] Fixing links for checkpoints & savepoints in the first …
vintageplayer commented on pull request #13951: URL: https://github.com/apache/flink/pull/13951#issuecomment-722862235 @rmetzger Have updated the pull request description. Please take a look. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13942: [FLINK-19990][table-planner-blink] MultipleInputNodeCreationProcessor#isChainableSource now considers DataStreamScanProvider
flinkbot edited a comment on pull request #13942: URL: https://github.com/apache/flink/pull/13942#issuecomment-722358038 ## CI report: * 30b3bcd2da00b89794fe83bee5c5c0113949bbfa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9110) * bdf73f2711715bf90e19bd8387ee86e15216dabb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9135) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13956: [FLINK-20006] Fix the unstable FileSinkITCase
flinkbot commented on pull request #13956: URL: https://github.com/apache/flink/pull/13956#issuecomment-722868816 ## CI report: * 1de2eca68fe489aadb55ed3084d18f4cd7ea723f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13864: [FLINK-19543][k8s] Implement RunningJobsRegistry, JobGraphStore based on Kubernetes API
flinkbot edited a comment on pull request #13864: URL: https://github.com/apache/flink/pull/13864#issuecomment-719924065 ## CI report: * f5ee007a4e17a98cc6252e0f14c023d657b9438a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9089) * 87a51133e4aaa9b4a6867686868d5467b91add6f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9137) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13897: [FLINK-19939][table-planner-blink] Remove redundant union from multiple input node
flinkbot edited a comment on pull request #13897: URL: https://github.com/apache/flink/pull/13897#issuecomment-720935204 ## CI report: * 6ee9a3823a7b502c7c07daa304148ae60f42d237 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9078) * 35c3c55c0dc376027cc9a4a89001d13e0620a8ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13930: [FLINK-19974][python][e2e] Fix the bug that kafka related services are not teardown properly after tests finished
flinkbot edited a comment on pull request #13930: URL: https://github.com/apache/flink/pull/13930#issuecomment-722114476 ## CI report: * 1b07904cff7c6401452f07dfc88775d0d3f84853 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9036) * a182d1ecd088484bb971f6cb63eb49213fd3cd3b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13956: [FLINK-20006] Fix the unstable FileSinkITCase
flinkbot edited a comment on pull request #13956: URL: https://github.com/apache/flink/pull/13956#issuecomment-722868816 ## CI report: * 1de2eca68fe489aadb55ed3084d18f4cd7ea723f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9138) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13931: [FLINK-19811][table-planner] Introduce RexSimplify to simplify SEARCH conditions
flinkbot edited a comment on pull request #13931: URL: https://github.com/apache/flink/pull/13931#issuecomment-722165982 ## CI report: * 83c419b7fbf59830387cdac4418e63c4ae1b3296 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9054) * df595b51b06c543ef0c29454343d10b4e110bb3d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518547824
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCompletedCheckpointStore.java
##
@@ -118,33 +112,22 @@ public boolean requiresExternalizedCheckpoints() {
}
/**
-* Gets the latest checkpoint from ZooKeeper and removes all others.
-*
-* Important: Even if there are more than one
checkpoint in ZooKeeper,
-* this will only recover the latest and discard the others. Otherwise,
there is no guarantee
-* that the history of checkpoints is consistent.
+* Recover all the valid checkpoints from state handle store. All the
successfully recovered checkpoints will
+* be added to {@link #completedCheckpoints} sorted by checkpoint id.
*/
@Override
public void recover() throws Exception {
- LOG.info("Recovering checkpoints from ZooKeeper.");
+ LOG.info("Recovering checkpoints from {}.",
checkpointStateHandleStore);
// Get all there is first
- List,
String>> initialCheckpoints;
- while (true) {
- try {
- initialCheckpoints =
checkpointsInZooKeeper.getAll();
- break;
- }
- catch (ConcurrentModificationException e) {
- LOG.warn("Concurrent modification while reading
from ZooKeeper. Retrying.");
- }
- }
+ final List,
String>> initialCheckpoints =
+ checkpointStateHandleStore.getAll();
Review comment:
1. IIUC, we will never throw a `ConcurrentModificationException` in
`StateHandleStore#getAllAndLock`. So this retry does not take any effect.
2. For `ZooKeeperStateHandleStore` implementation, it could retry internally
to avoid concurrent modifications.
3. For `KubernetesStateHandleStore` implementation, we do not need retry
since we could get all the data in one time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dianfu closed pull request #13935: [FLINK-18926][python][doc] Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section.
dianfu closed pull request #13935: URL: https://github.com/apache/flink/pull/13935 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18926) Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section
[ https://issues.apache.org/jira/browse/FLINK-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18926: Component/s: API / Python > Add a "Environment Variables" document under the "Python API" -> "User > Guide" -> "Table API" section > - > > Key: FLINK-18926 > URL: https://issues.apache.org/jira/browse/FLINK-18926 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Documentation >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18926) Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section
[ https://issues.apache.org/jira/browse/FLINK-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-18926. --- Fix Version/s: 1.12.0 Resolution: Fixed Merged to master via e0a0aa606072e48018c178c1c3f47557f445e5ef > Add a "Environment Variables" document under the "Python API" -> "User > Guide" -> "Table API" section > - > > Key: FLINK-18926 > URL: https://issues.apache.org/jira/browse/FLINK-18926 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18929) Add a "API Docs" link (linked to the generated sphinx docs) under the "Python API" section
[ https://issues.apache.org/jira/browse/FLINK-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-18929. --- Assignee: (was: Huang Xingbo) Resolution: Abandoned [~hxbks2ks] Agree with you. Let's close this ticket for now. > Add a "API Docs" link (linked to the generated sphinx docs) under the "Python > API" section > -- > > Key: FLINK-18929 > URL: https://issues.apache.org/jira/browse/FLINK-18929 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Wei Zhong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on a change in pull request #13871: [FLINK-19544][k8s] Implement CheckpointRecoveryFactory based on Kubernetes API
wangyang0918 commented on a change in pull request #13871:
URL: https://github.com/apache/flink/pull/13871#discussion_r518550184
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/TestingRetrievableStateStorageHelper.java
##
@@ -41,22 +64,43 @@
private final T state;
- private TestingRetrievableStateHandle(T state) {
+ private FunctionWithException
retrieveStateFunction;
+
+ private RunnableWithException discardStateRunnable;
+
+ private Function getStateSizeFunction;
+
+ private TestingRetrievableStateHandle(
+ T state,
+ FunctionWithException
retrieveStateFunction,
+ RunnableWithException discardStateRunnable,
+ Function getStateSizeFunction) {
this.state = state;
+ this.retrieveStateFunction = retrieveStateFunction;
Review comment:
I am not fully understand here. Do you mean combining
`retrieveStateFunction` and `getStateSizeFunction`? If it is, we have two
question here.
* How to get the size of retrieved `state`?
* `getStateSize` do not throw any exception but `retrieveState` will.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] danny0405 commented on pull request #13865: [FLINK-19906][table-planner] Fix incorrect result when compare two binary fields
danny0405 commented on pull request #13865: URL: https://github.com/apache/flink/pull/13865#issuecomment-722905527 Sure, let me take a look ~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13931: [FLINK-19811][table-planner] Introduce RexSimplify to simplify SEARCH conditions
flinkbot edited a comment on pull request #13931: URL: https://github.com/apache/flink/pull/13931#issuecomment-722165982 ## CI report: * 83c419b7fbf59830387cdac4418e63c4ae1b3296 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9054) * df595b51b06c543ef0c29454343d10b4e110bb3d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9141) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13942: [FLINK-19990][table-planner-blink] MultipleInputNodeCreationProcessor#isChainableSource now considers DataStreamScanProvider
flinkbot edited a comment on pull request #13942: URL: https://github.com/apache/flink/pull/13942#issuecomment-722358038 ## CI report: * bdf73f2711715bf90e19bd8387ee86e15216dabb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9135) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13897: [FLINK-19939][table-planner-blink] Remove redundant union from multiple input node
flinkbot edited a comment on pull request #13897: URL: https://github.com/apache/flink/pull/13897#issuecomment-720935204 ## CI report: * 6ee9a3823a7b502c7c07daa304148ae60f42d237 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9078) * 35c3c55c0dc376027cc9a4a89001d13e0620a8ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9139) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13930: [FLINK-19974][python][e2e] Fix the bug that kafka related services are not teardown properly after tests finished
flinkbot edited a comment on pull request #13930: URL: https://github.com/apache/flink/pull/13930#issuecomment-722114476 ## CI report: * 1b07904cff7c6401452f07dfc88775d0d3f84853 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9036) * a182d1ecd088484bb971f6cb63eb49213fd3cd3b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9140) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518555888
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##
@@ -64,81 +61,64 @@
/**
* Snapshot from the {@link
org.apache.flink.runtime.state.OperatorStateBackend}.
*/
- @Nonnull
private final StateObjectCollection
managedOperatorState;
/**
* Snapshot written using {@link
org.apache.flink.runtime.state.OperatorStateCheckpointOutputStream}.
*/
- @Nonnull
private final StateObjectCollection
rawOperatorState;
/**
* Snapshot from {@link
org.apache.flink.runtime.state.KeyedStateBackend}.
*/
- @Nonnull
private final StateObjectCollection managedKeyedState;
/**
* Snapshot written using {@link
org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream}.
*/
- @Nonnull
private final StateObjectCollection rawKeyedState;
- @Nonnull
private final StateObjectCollection
inputChannelState;
- @Nonnull
private final StateObjectCollection
resultSubpartitionState;
/**
-* The state size. This is also part of the deserialized state handle.
-* We store it here in order to not deserialize the state handle when
-* gathering stats.
+* The subpartitions mappings per partition set when the output
operator for a partition was rescaled. The key is
+* the partition id and the value contains all subtask indexes of the
output operator before rescaling.
*/
- private final long stateSize;
+ private final InflightDataRescalingDescriptor inputRescalingDescriptor;
Review comment:
Added a comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] AHeise commented on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.
AHeise commented on pull request #13735: URL: https://github.com/apache/flink/pull/13735#issuecomment-722914106 > Thanks for updating the PR, LGTM. > > I'm only concerned about changing the partitioner upon restart. For example, if the job is modified so that DOP of upstream is no longer equal to DOP of the downstream; then it switches from `Forward` to `Rebalance` IIUC. Which will break state distribution, so it should be prohibited. > If that's a valid concern I'd approach it in a different effort. The concern is valid. I'm adding a check in the second PR then. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20015) Failure to execute streaming query
Satyam Shekhar created FLINK-20015: -- Summary: Failure to execute streaming query Key: FLINK-20015 URL: https://issues.apache.org/jira/browse/FLINK-20015 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.11.1 Reporter: Satyam Shekhar Hello, I have a table T0 with the following schema - root |-- amount: BIGINT |-- timestamp: TIMESTAMP(3) The following two queries fail execution on the above table when executed in streaming mode using the Blink planner. WITH A AS ( SELECT COUNT(*) AS ct, tumble_end(`timestamp`, INTERVAL '1' MINUTE) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm WITH A AS ( SELECT COUNT(*) AS ct, tumble_rowtime(`timestamp`, INTERVAL '1' MINUTE) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm The two queries are very similar and only differ in their use of tumble_end and tumble_rowtime operator. Both queries use timestamp column as their rowtime attribute. Casting "tm" column to timestamp makes both queries work - WITH A AS ( SELECT COUNT(*) AS ct, CAST(tumble_end(`timestamp`, INTERVAL '1' MINUTE) as TIMESTAMP(3)) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm This workaround, however, loses the rowtime attribute from the output resultset for the second query. The first query fails with the following exception - java.lang.RuntimeException: class java.sql.Timestamp cannot be cast to class java.lang.Long (java.sql.Timestamp is in module java.sql of loader 'platform'; java.lang.Long is in module java.base of loader 'bootstrap') at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at SinkConversion$166.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at StreamExecCalc$163.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.outputNullPadding(StreamingJoinOperator.java:314) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement(StreamingJoinOperator.java:206) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement1(StreamingJoinOperator.java:115) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processRecord1(StreamTwoInputProcessor.java:132) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.lambda$new$0(StreamTwoInputProcessor.java:99) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessor.java:364) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:179) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:345) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:191) at org.apache.fl
[jira] [Created] (FLINK-20016) Support TimestampAssigner and WatermarkGenerator for Python DataStream API.
Shuiqiang Chen created FLINK-20016: -- Summary: Support TimestampAssigner and WatermarkGenerator for Python DataStream API. Key: FLINK-20016 URL: https://issues.apache.org/jira/browse/FLINK-20016 Project: Flink Issue Type: Improvement Components: API / Python Reporter: Shuiqiang Chen Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518555888
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##
@@ -64,81 +61,64 @@
/**
* Snapshot from the {@link
org.apache.flink.runtime.state.OperatorStateBackend}.
*/
- @Nonnull
private final StateObjectCollection
managedOperatorState;
/**
* Snapshot written using {@link
org.apache.flink.runtime.state.OperatorStateCheckpointOutputStream}.
*/
- @Nonnull
private final StateObjectCollection
rawOperatorState;
/**
* Snapshot from {@link
org.apache.flink.runtime.state.KeyedStateBackend}.
*/
- @Nonnull
private final StateObjectCollection managedKeyedState;
/**
* Snapshot written using {@link
org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream}.
*/
- @Nonnull
private final StateObjectCollection rawKeyedState;
- @Nonnull
private final StateObjectCollection
inputChannelState;
- @Nonnull
private final StateObjectCollection
resultSubpartitionState;
/**
-* The state size. This is also part of the deserialized state handle.
-* We store it here in order to not deserialize the state handle when
-* gathering stats.
+* The subpartitions mappings per partition set when the output
operator for a partition was rescaled. The key is
+* the partition id and the value contains all subtask indexes of the
output operator before rescaling.
*/
- private final long stateSize;
+ private final InflightDataRescalingDescriptor inputRescalingDescriptor;
Review comment:
Added a comment.
```
Note that this field is only set by {@link StateAssignmentOperation} and
will not be persisted in the checkpoint itself as it can only be calculated if
the the post-recovery scale factor is known.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Myasuka merged pull request #13587: [FLINK-18004][docs] Update checkpoint UI related pictures in documentation
Myasuka merged pull request #13587: URL: https://github.com/apache/flink/pull/13587 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20015) Failure to execute streaming query
[
https://issues.apache.org/jira/browse/FLINK-20015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satyam Shekhar updated FLINK-20015:
---
Description:
Hello,
I have a table T0 with the following schema -
{{root}}
{{ |-- amount: BIGINT}}
{{ |-- timestamp: TIMESTAMP(3)}}
{{ }}
{{ The following two queries fail execution on the above table when executed in
streaming mode using the Blink planner.}}
{{ }}
{{ WITH A AS (}}
{{ SELECT COUNT(*) AS ct, tumble_end(`timestamp`, INTERVAL '1' MINUTE) as tm}}
{{ FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE))}}
{{ select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm}}
{{ }}
{{ WITH A AS (}}
{{ SELECT COUNT(*) AS ct, tumble_rowtime(`timestamp`, INTERVAL '1' MINUTE) as
tm}}
{{ FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE))}}
{{ select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm}}
{{ }}
{{ The two queries are very similar and only differ in their use of tumble_end
and tumble_rowtime operator. Both queries use the timestamp column as their
rowtime attribute. Casting "tm" column to timestamp makes both queries work -}}
{{ }}
{{ WITH A AS (}}
{{ SELECT COUNT(*) AS ct, CAST(tumble_end(`timestamp`, INTERVAL '1' MINUTE)
as TIMESTAMP(3)) as tm}}
{{ FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE))}}
{{ select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm}}
{{ }}
{{ This workaround, however, loses the rowtime attribute from the output
resultset for the second query.}}
{{ }}
{{ The first query fails with the following exception -}}
{{ }}
{{ java.lang.RuntimeException: class java.sql.Timestamp cannot be cast to class
java.lang.Long (java.sql.Timestamp is in module java.sql of loader 'platform';
java.lang.Long is in module java.base of loader 'bootstrap')}}
{{ at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)}}
{{ at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)}}
{{ at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)}}
{{ at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)}}
{{ at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)}}
{{ at SinkConversion$166.processElement(Unknown Source)}}
{{ at
org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626)}}
{{ at
org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603)}}
{{ at
org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563)}}
{{ at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)}}
{{ at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)}}
{{ at StreamExecCalc$163.processElement(Unknown Source)}}
{{ at
org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626)}}
{{ at
org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603)}}
{{ at
org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563)}}
{{ at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)}}
{{ at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)}}
{{ at
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53)}}
{{ at
org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.outputNullPadding(StreamingJoinOperator.java:314)}}
{{ at
org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement(StreamingJoinOperator.java:206)}}
{{ at
org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement1(StreamingJoinOperator.java:115)}}
{{ at
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processRecord1(StreamTwoInputProcessor.java:132)}}
{{ at
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.lambda$new$0(StreamTwoInputProcessor.java:99)}}
{{ at
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessor.java:364)}}
{{ at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178)}}
{{ at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153)}}
{{ at
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:179)}}
{{ at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:345)}}
{{ at
org.apache.flink.streaming.runtime.tasks.mail
[jira] [Updated] (FLINK-20015) Failure to execute streaming query
[ https://issues.apache.org/jira/browse/FLINK-20015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satyam Shekhar updated FLINK-20015: --- Description: Hello, I have a table T0 with the following schema - root |-- amount: BIGINT |-- timestamp: TIMESTAMP(3) The following two queries fail execution on the above table when executed in streaming mode using the Blink planner. WITH A AS ( SELECT COUNT(*) AS ct, tumble_end(`timestamp`, INTERVAL '1' MINUTE) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm WITH A AS ( SELECT COUNT(*) AS ct, tumble_rowtime(`timestamp`, INTERVAL '1' MINUTE) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm The two queries are very similar and only differ in their use of tumble_end and tumble_rowtime operator. Both queries use the timestamp column as their rowtime attribute. Casting "tm" column to timestamp makes both queries work - WITH A AS ( SELECT COUNT(*) AS ct, CAST(tumble_end(`timestamp`, INTERVAL '1' MINUTE) as TIMESTAMP(3)) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm This workaround, however, loses the rowtime attribute from the output resultset for the second query. The first query fails with the following exception - java.lang.RuntimeException: class java.sql.Timestamp cannot be cast to class java.lang.Long (java.sql.Timestamp is in module java.sql of loader 'platform'; java.lang.Long is in module java.base of loader 'bootstrap') at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at SinkConversion$166.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at StreamExecCalc$163.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.outputNullPadding(StreamingJoinOperator.java:314) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement(StreamingJoinOperator.java:206) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement1(StreamingJoinOperator.java:115) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processRecord1(StreamTwoInputProcessor.java:132) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.lambda$new$0(StreamTwoInputProcessor.java:99) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessor.java:364) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:179) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:345) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:191) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181) at org.apache.flink.streaming.runtime.tasks.Strea
[jira] [Updated] (FLINK-20015) Failure to execute streaming query
[ https://issues.apache.org/jira/browse/FLINK-20015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satyam Shekhar updated FLINK-20015: --- Description: Hello, I have a table T0 with the following schema - root |-- amount: BIGINT |-- timestamp: TIMESTAMP(3) The following two queries fail execution on the above table when executed in streaming mode using the Blink planner. WITH A AS ( SELECT COUNT(\*) AS ct, tumble_end(`timestamp`, INTERVAL '1' MINUTE) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm WITH A AS ( SELECT COUNT(\*) AS ct, tumble_rowtime(`timestamp`, INTERVAL '1' MINUTE) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm The two queries are very similar and only differ in their use of tumble_end and tumble_rowtime operator. Both queries use the timestamp column as their rowtime attribute. Casting "tm" column to timestamp makes both queries work - WITH A AS ( SELECT COUNT(\*) AS ct, CAST(tumble_end(`timestamp`, INTERVAL '1' MINUTE) as TIMESTAMP(3)) as tm FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm This workaround, however, loses the rowtime attribute from the output resultset for the second query. The first query fails with the following exception - java.lang.RuntimeException: class java.sql.Timestamp cannot be cast to class java.lang.Long (java.sql.Timestamp is in module java.sql of loader 'platform'; java.lang.Long is in module java.base of loader 'bootstrap') at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at SinkConversion$166.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at StreamExecCalc$163.processElement(Unknown Source) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.outputNullPadding(StreamingJoinOperator.java:314) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement(StreamingJoinOperator.java:206) at org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement1(StreamingJoinOperator.java:115) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processRecord1(StreamTwoInputProcessor.java:132) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.lambda$new$0(StreamTwoInputProcessor.java:99) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessor.java:364) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178) at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153) at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:179) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:345) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:191) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181) at org.apache.flink.streaming.runtime.tasks.S
[GitHub] [flink] AHeise merged pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.
AHeise merged pull request #13735: URL: https://github.com/apache/flink/pull/13735 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19533) Add state assignment to channel state during rescaling
[ https://issues.apache.org/jira/browse/FLINK-19533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227203#comment-17227203 ] Arvid Heise commented on FLINK-19533: - Merged into master as a2267fa28a44c4cf4cc906859c5531c94e994bac . > Add state assignment to channel state during rescaling > -- > > Key: FLINK-19533 > URL: https://issues.apache.org/jira/browse/FLINK-19533 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Affects Versions: 1.12.0 >Reporter: Arvid Heise >Assignee: Arvid Heise >Priority: Major > Labels: pull-request-available > > Enhance StateAssignmentOperation to create a channel state mapping during > rescaling, where channels before rescaling are mapped to channels after > rescaling. > Example: > Before rescaling: parallelism of 2, channel 0 = key group range [0,64), > channel 1 = key group range [64,128) > After rescaling: parallelism of 3, channel 0 = key group range [0,43), > channel 1 = key group range [43,86), channel 2 = key group range [86,128) > Mapping: 0 -> [0, 1], 1 -> [1, 2] > > The mapping is passed to the respective channels through deployment > descriptors. It will be used in a separate ticket to restore the correct > buffers from pre-rescaling channels to post-rescaling channels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-19533) Add state assignment to channel state during rescaling
[ https://issues.apache.org/jira/browse/FLINK-19533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-19533. - Fix Version/s: 1.12.0 Resolution: Fixed > Add state assignment to channel state during rescaling > -- > > Key: FLINK-19533 > URL: https://issues.apache.org/jira/browse/FLINK-19533 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Affects Versions: 1.12.0 >Reporter: Arvid Heise >Assignee: Arvid Heise >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Enhance StateAssignmentOperation to create a channel state mapping during > rescaling, where channels before rescaling are mapped to channels after > rescaling. > Example: > Before rescaling: parallelism of 2, channel 0 = key group range [0,64), > channel 1 = key group range [64,128) > After rescaling: parallelism of 3, channel 0 = key group range [0,43), > channel 1 = key group range [43,86), channel 2 = key group range [86,128) > Mapping: 0 -> [0, 1], 1 -> [1, 2] > > The mapping is passed to the respective channels through deployment > descriptors. It will be used in a separate ticket to restore the correct > buffers from pre-rescaling channels to post-rescaling channels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TsReaper commented on pull request #13942: [FLINK-19990][table-planner-blink] MultipleInputNodeCreationProcessor#isChainableSource now considers DataStreamScanProvider
TsReaper commented on pull request #13942: URL: https://github.com/apache/flink/pull/13942#issuecomment-722916753 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19665) AlternatingCheckpointBarrierHandlerTest.testMetricsAlternation unstable
[
https://issues.apache.org/jira/browse/FLINK-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-19665.
--
Fix Version/s: 1.12.0
Resolution: Fixed
Yes, that's right. Thanks for pointing this out [~kezhuw]
> AlternatingCheckpointBarrierHandlerTest.testMetricsAlternation unstable
> ---
>
> Key: FLINK-19665
> URL: https://issues.apache.org/jira/browse/FLINK-19665
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8479&view=logs&j=6e58d712-c5cc-52fb-0895-6ff7bd56c46b&t=f30a8e80-b2cf-535c-9952-7f521a4ae374
> {code}
> [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.115
> s <<< FAILURE! - in
> org.apache.flink.streaming.runtime.io.AlternatingCheckpointBarrierHandlerTest
> [ERROR]
> testMetricsAlternation(org.apache.flink.streaming.runtime.io.AlternatingCheckpointBarrierHandlerTest)
> Time elapsed: 0.017 s <<< FAILURE!
> java.lang.AssertionError:
> Expected: a value less than or equal to <74001L>
> but: <137102L> was greater than <74001L>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> at
> org.apache.flink.streaming.runtime.io.AlternatingCheckpointBarrierHandlerTest.assertMetrics(AlternatingCheckpointBarrierHandlerTest.java:212)
> at
> org.apache.flink.streaming.runtime.io.AlternatingCheckpointBarrierHandlerTest.testMetricsAlternation(AlternatingCheckpointBarrierHandlerTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13942: [FLINK-19990][table-planner-blink] MultipleInputNodeCreationProcessor#isChainableSource now considers DataStreamScanProvider
flinkbot edited a comment on pull request #13942: URL: https://github.com/apache/flink/pull/13942#issuecomment-722358038 ## CI report: * bdf73f2711715bf90e19bd8387ee86e15216dabb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9135) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9144) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20015) Failure to execute streaming SQL query
[ https://issues.apache.org/jira/browse/FLINK-20015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satyam Shekhar updated FLINK-20015: --- Summary: Failure to execute streaming SQL query (was: Failure to execute streaming query) > Failure to execute streaming SQL query > -- > > Key: FLINK-20015 > URL: https://issues.apache.org/jira/browse/FLINK-20015 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.11.1 >Reporter: Satyam Shekhar >Priority: Major > > Hello, > > I have a table T0 with the following schema - > root > |-- amount: BIGINT > |-- timestamp: TIMESTAMP(3) > > The following two queries fail execution on the above table when executed in > streaming mode using the Blink planner. > > WITH A AS ( > SELECT COUNT(\*) AS ct, tumble_end(`timestamp`, INTERVAL '1' MINUTE) as tm > FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) > select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm > > WITH A AS ( > SELECT COUNT(\*) AS ct, tumble_rowtime(`timestamp`, INTERVAL '1' MINUTE) > as tm > FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) > select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm > > The two queries are very similar and only differ in their use of tumble_end > and tumble_rowtime operator. Both queries use the timestamp column as their > rowtime attribute. Casting "tm" column to timestamp makes both queries work - > > WITH A AS ( > SELECT COUNT(\*) AS ct, CAST(tumble_end(`timestamp`, INTERVAL '1' MINUTE) > as TIMESTAMP(3)) as tm > FROM T0 GROUP BY tumble(`timestamp`, INTERVAL '1' MINUTE)) > select L.ct, R.ct, L.tm, R.tm from A as L left join A as R on L.tm = R.tm > > This workaround, however, loses the rowtime attribute from the output > resultset for the second query. > > The first query fails with the following exception - > > java.lang.RuntimeException: class java.sql.Timestamp cannot be cast to class > java.lang.Long (java.sql.Timestamp is in module java.sql of loader > 'platform'; java.lang.Long is in module java.base of loader 'bootstrap') > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110) > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89) > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) > at SinkConversion$166.processElement(Unknown Source) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) > at StreamExecCalc$163.processElement(Unknown Source) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:626) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:603) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:563) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) > at > org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53) > at > org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.outputNullPadding(StreamingJoinOperator.java:314) > at > org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement(StreamingJoinOperator.java:206) > at > org.apache.flink.table.runtime.operators.join.stream.StreamingJoinOperator.processElement1(StreamingJoinOperator.java:115) > at > org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processRecord1(StreamTwoInputProcessor.java:132) > at > org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.lambda$new$0(StreamTwoInputProcessor.java:99) > at > org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessor.java:364) > at > org.apache.flink.streaming.runtime.io.StreamTaskNetworkIn
[jira] [Commented] (FLINK-19956) $ does not work on variables without being qualified in Scala
[
https://issues.apache.org/jira/browse/FLINK-19956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227219#comment-17227219
]
Timo Walther commented on FLINK-19956:
--
Thanks for the research [~jark]. I forgot that we fixed that "recently" for
1.12. To summarize this issue a workaround for 1.11 is:
{code}
val column = "f0"
table.filter($"$column" === true)
{code}
In 1.11 the $() method was not available in Scala.
> $ does not work on variables without being qualified in Scala
> -
>
> Key: FLINK-19956
> URL: https://issues.apache.org/jira/browse/FLINK-19956
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.11.2
> Environment: MacOS
>Reporter: Rex Remind
>Priority: Minor
>
> This does not compile:
> {code:java}
> val columnName = "bool_column"
> table.filter($(column) === true) {code}
>
> {color:#33}This does:{color}
> {code:java}
> val columnName = "bool_column"
> table.filter(Expressions.$(column) === true) {code}
>
> {color:#33}There's nothing obviously documented to using the later.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19687) Support to get execution plan in `StatementSet`
[ https://issues.apache.org/jira/browse/FLINK-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaozilong updated FLINK-19687: --- Description: Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet exception "No operators defined in streaming topology. Cannot execute." when use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I found translation operations only happen when StatementSet.execute() is called in Flink SQL 1.11. So we cannot get job's execution plan before the job submit? Can we support to get execution plan in `StatementSet`? was:Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet exception "No operators defined in streaming topology. Cannot execute." when use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I found translation operations only happen when StatementSet.execute() is called in Flink SQL 1.11. So we cannot get job's execution plan before the job submit? Can we support to get execution plan in `StatementSet` or invoke method `TableEnvironmentImpl#translate` in `StatementSetImpl#addInsertSql`? I think the latter is better so that we can reuse `env.getExecutionPlan()`. > Support to get execution plan in `StatementSet` > --- > > Key: FLINK-19687 > URL: https://issues.apache.org/jira/browse/FLINK-19687 > Project: Flink > Issue Type: Wish > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: xiaozilong >Priority: Major > > Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet > exception "No operators defined in streaming topology. Cannot execute." when > use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I > found translation operations only happen when StatementSet.execute() is > called in Flink SQL 1.11. So we cannot get job's execution plan before the > job submit? Can we support to get execution plan in `StatementSet`? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18926) Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section
[ https://issues.apache.org/jira/browse/FLINK-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18926: Fix Version/s: 1.11.3 > Add a "Environment Variables" document under the "Python API" -> "User > Guide" -> "Table API" section > - > > Key: FLINK-18926 > URL: https://issues.apache.org/jira/browse/FLINK-18926 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Documentation >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18926) Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section
[ https://issues.apache.org/jira/browse/FLINK-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227191#comment-17227191 ] Dian Fu edited comment on FLINK-18926 at 11/6/20, 7:39 AM: --- Merged to - master via e0a0aa606072e48018c178c1c3f47557f445e5ef - release-1.11 via 42039658a832d7cc22e9aa3fb2427c765469b8e7 was (Author: dian.fu): Merged to master via e0a0aa606072e48018c178c1c3f47557f445e5ef > Add a "Environment Variables" document under the "Python API" -> "User > Guide" -> "Table API" section > - > > Key: FLINK-18926 > URL: https://issues.apache.org/jira/browse/FLINK-18926 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Documentation >Reporter: Wei Zhong >Assignee: Wei Zhong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13942: [FLINK-19990][table-planner-blink] MultipleInputNodeCreationProcessor#isChainableSource now considers DataStreamScanProvider
flinkbot edited a comment on pull request #13942: URL: https://github.com/apache/flink/pull/13942#issuecomment-722358038 ## CI report: * bdf73f2711715bf90e19bd8387ee86e15216dabb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9144) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9135) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13930: [FLINK-19974][python][e2e] Fix the bug that kafka related services are not teardown properly after tests finished
flinkbot edited a comment on pull request #13930: URL: https://github.com/apache/flink/pull/13930#issuecomment-722114476 ## CI report: * 1b07904cff7c6401452f07dfc88775d0d3f84853 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9036) * a182d1ecd088484bb971f6cb63eb49213fd3cd3b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9140) * f826010f2bde1f2326044d37a616d620252703b9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on a change in pull request #13939: [FLINK-19992][hive] Integrate new orc to Hive source
lirui-apache commented on a change in pull request #13939:
URL: https://github.com/apache/flink/pull/13939#discussion_r518573163
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HiveBulkFormatAdapter.java
##
@@ -107,24 +114,78 @@ public boolean isSplittable() {
return InternalTypeInfo.of(producedRowType);
}
+ private RowType tableRowType() {
+ LogicalType[] types =
Arrays.stream(fieldTypes).map(DataType::getLogicalType).toArray(LogicalType[]::new);
+ return RowType.of(types, fieldNames);
+ }
+
+ private int[] projectedFields() {
Review comment:
We already have a `computeSelectedFields` method, can it be reused?
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HiveBulkFormatAdapter.java
##
@@ -107,24 +114,78 @@ public boolean isSplittable() {
return InternalTypeInfo.of(producedRowType);
}
+ private RowType tableRowType() {
+ LogicalType[] types =
Arrays.stream(fieldTypes).map(DataType::getLogicalType).toArray(LogicalType[]::new);
+ return RowType.of(types, fieldNames);
+ }
+
+ private int[] projectedFields() {
+ List nameList = Arrays.asList(fieldNames);
+ int[] projectedFields = new
int[producedRowType.getFieldCount()];
+ List projectedNames = producedRowType.getFieldNames();
+ for (int i = 0; i < projectedFields.length; i++) {
+ projectedFields[i] =
nameList.indexOf(projectedNames.get(i));
+ }
+ return projectedFields;
+ }
+
private BulkFormat
createBulkFormatForSplit(HiveSourceSplit split) {
if (!useMapRedReader &&
useParquetVectorizedRead(split.getHiveTablePartition())) {
- PartitionFieldExtractor extractor =
(PartitionFieldExtractor)
- (split1, fieldName, fieldType) ->
split1.getHiveTablePartition().getPartitionSpec().get(fieldName);
return
ParquetColumnarRowInputFormat.createPartitionedFormat(
jobConfWrapper.conf(),
producedRowType,
partitionKeys,
- extractor,
+ PARTITION_FIELD_EXTRACTOR,
DEFAULT_SIZE,
hiveVersion.startsWith("3"),
false
);
+ } else if (!useMapRedReader &&
useOrcVectorizedRead(split.getHiveTablePartition())) {
+ return createOrcFormat();
} else {
return new HiveMapRedBulkFormat();
}
}
+ private OrcColumnarRowFileInputFormat
createOrcFormat() {
+ return hiveVersion.startsWith("1.") ?
+
OrcColumnarRowFileInputFormat.createPartitionedFormat(
+ OrcShim.createShim(hiveVersion),
+ jobConfWrapper.conf(),
+ tableRowType(),
+ partitionKeys,
+ PARTITION_FIELD_EXTRACTOR,
+ projectedFields(),
+ new ArrayList<>(),
Review comment:
`Collections.emptyList()`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] danny0405 commented on a change in pull request #13865: [FLINK-19906][table-planner] Fix incorrect result when compare two binary fields
danny0405 commented on a change in pull request #13865:
URL: https://github.com/apache/flink/pull/13865#discussion_r518578072
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/functions/SqlFunctionUtils.java
##
@@ -1116,4 +1116,26 @@ public static float struncate(float b0) {
public static float struncate(float b0, int b1) {
return (float) doubleValue(struncate(castFrom((double) b0, 38,
18), b1));
}
+
+ /**
+* Compare between two byte arrays.
+* The result is 1 if array1 is great than array2,
+* -1 if array1 is less than array2 and 0 if array1 is equal array2.
+* Note: Currently, this is used in {@code ScalarOperatorGens}
+* for comparing two fields of binary or varbinary type.
+*
+* @param array1 byte array1 to be compared.
+* @param array2 byte array2 to be compared.
+* @return Target value.
+*/
+ public static int byteArrayCompare(byte[] array1, byte[] array2) {
Review comment:
```java
/**
* Compares two byte arrays in lexicographical order.
*
* The result is
* positive if {@code array1} is great than {@code array2},
* negative if {@code array1} is less than {@code array2} and
* 0 if {@code array1} is equal to {@code array2}.
*
* Note: Currently, this is used in {@code ScalarOperatorGens}
* for comparing two fields of binary or varbinary type.
*
* @param array1 byte array to compare.
* @param array2 byte array to compare.
* @return an Integer indicating which one is bigger
*/
```
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/expressions/ScalarOperatorsTest.scala
##
@@ -62,6 +62,40 @@ class ScalarOperatorsTest extends ScalarOperatorsTestBase {
)
}
+ @Test
+ def testCompareOperator(): Unit= {
+
+// f18 and f19 have same length.
+testSqlApi(
+ "f18 > f19",
+ "true")
Review comment:
We may also need to add some test in
`SqlExpressionTest.testComparisonFunctions`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-14431) Update TaskManager's memory information to match its memory composition
[
https://issues.apache.org/jira/browse/FLINK-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-14431:
-
Release Note:
FLINK-14431 (FLIP-102) updated the metrics overview page of the TaskManagers
aligning it with the memory model introduced in FLINK-13980 (FLIP-49).
As part of this work, new metrics have been introduced for managed memory,
netty shuffle memory (network) and metaspace. Minor naming bugs have been
corrected as well.
was:
FLINK-14431 (FLIP-102) updated the metrics overview page of the TaskManagers
aligning it with the memory model introduced in FLIP-49.
As part of this work, new metrics have been introduced for managed memory,
netty shuffle memory (network) and metaspace. Minor naming bugs have been
corrected as well.
> Update TaskManager's memory information to match its memory composition
> ---
>
> Key: FLINK-14431
> URL: https://issues.apache.org/jira/browse/FLINK-14431
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST, Runtime / Task, Runtime / Web Frontend
>Reporter: lining
>Assignee: Matthias
>Priority: Major
> Attachments: image-2019-10-17-17-58-50-342.png,
> image-2019-10-17-18-01-09-353.png, image-2019-10-17-18-29-53-329.png,
> image-2019-10-24-16-19-15-499.png, image-2019-10-24-16-20-23-210.png,
> image-2019-10-24-16-22-27-360.png, image-2019-12-19-18-09-05-542.png,
> image-2019-12-19-18-27-38-589.png, image-2019-12-19-18-28-01-447.png
>
>
> h3. Motivation
> There are several shortcomings of current (Flink 1.10) Flink TaskManager
> memory information show in rest api.
> h4. (1) The information from HardwareDescription is difficult to match the
> memory compositions of TaskManager in flip-49. As below picture show：
> !image-2019-12-19-18-09-05-542.png|width=444,height=389!
> * what's the meaning of HardwareDescription.sizeOfJvmHeap.
> * the user couldn't get resource config about TaskManager.
> h4. (2) There isn't information for managed memory.
> * no metric for managed memory.
> h4. (3) There isn't information for shuffle memory
> * according to TaskManagerMetricsInfo's memorySegmentsTotal(ps: shuffle
> segment total size), the user couldn't get shuffle memory.
> h4. (4) The metrics in the TaskManager's metrics page do not correspond to
> the resource configuration of taskmanager
> * It is difficult for users to update taskmanager's resource configuration
> based on metrics because users couldn’t find configuration items related to
> metrics.
> h3. Proposed Changes
> h4. Add TaskManageResourceInfo which match the memory compositions
> * information from TaskExecutorResourceSpec in flip-49, add it to
> TaskExecutorRegistration.
> {code:java}
> public class TaskManagerResourceInfo {
> private final double cpuCores;
> private final long frameworkHeap;
> private final long frameworkOffHeap;
> private final long taskHeap;
> private final long taskOffHeap;
> private final long shuffleMemory;
> private final long managedMemory;
> private final long jvmMetaSpace;
> private final long jvmOverhead;
> private final long totalProcessMemory;
> }
> {code}
> * url: /taskmanagers/:taskmanagerid
> * response: add
> {code:json}
> resource: {
> cpuCores: 4,
> frameworkHeap: 134217728,
> frameworkOffHeap: 134217728,
> taskHeap: 181193928,
> taskOffHeap: 0,
> shuffleMemory: 33554432,
> managedMemory: 322122552,
> jvmMetaSpace: 134217728,
> jvmOverhead: 134217728,
> totalProcessMemory: 1073741824
> }
> {code}
> h4. Add shuffle memory metric
> * add getTotalMemorySize and in getAvaliableMemorySize NetworkBufferPool
> {code:java}
> public long getTotalMemorySize() {
> return 1L * getTotalNumberOfMemorySegments() * memorySegmentSize;
> }
> public long getAvaliableMemorySize() {
> return 1L * getNumberOfAvailableMemorySegments() * memorySegmentSize;
> }{code}
> * update NettyShuffleMetricFactory#registerShuffleMetrics
> {code:java}
> private static final String METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY =
> "TotalMemoryCapacity";
> private static final String METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY =
> "AvaliableMemory";
> private static void registerShuffleMetrics(
> String groupName,
> MetricGroup metricGroup,
> NetworkBufferPool networkBufferPool) {
> MetricGroup networkGroup = metricGroup.addGroup(groupName);
> networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT,
>
> networkBufferPool::getTotalNumberOfMemorySegments);
> networkGroup. Gauge>gauge(METRIC_AVAILABLE_MEMORY_SEGMENT,
>
> networkBufferPool::getNumberOfAvailableMemorySegments);
> networkGroup. Gauge>gauge(METRIC_TOTAL_MEMORY_SEGMENT_TOTA
[GitHub] [flink] JingsongLi closed pull request #13915: [FLINK-19365][hive] Migrate Hive source to FLIP-27 source interface f…
JingsongLi closed pull request #13915: URL: https://github.com/apache/flink/pull/13915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19365) Migrate Hive source to FLIP-27 source interface for batch
[ https://issues.apache.org/jira/browse/FLINK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-19365. Resolution: Fixed master (1.12): 56207e8a578394c01022bbe848d03cf8b97c2380 > Migrate Hive source to FLIP-27 source interface for batch > - > > Key: FLINK-19365 > URL: https://issues.apache.org/jira/browse/FLINK-19365 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > I'll limit the scope here to batch reading, to make the PR easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi merged pull request #13915: [FLINK-19365][hive] Migrate Hive source to FLIP-27 source interface f…
JingsongLi merged pull request #13915: URL: https://github.com/apache/flink/pull/13915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13932: [FLINK-19947][Connectors / Common]Support sink parallelism configuration for Print connector
flinkbot edited a comment on pull request #13932: URL: https://github.com/apache/flink/pull/13932#issuecomment-722193609 ## CI report: * cd87ec7806bbe57badf0302188c7041a71349b59 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9049) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13931: [FLINK-19811][table-planner] Introduce RexSimplify to simplify SEARCH conditions
flinkbot edited a comment on pull request #13931: URL: https://github.com/apache/flink/pull/13931#issuecomment-722165982 ## CI report: * ea93667f09807f3e9eae819bed19678d7a699173 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9042) * 83c419b7fbf59830387cdac4418e63c4ae1b3296 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-14431) Update TaskManager's memory information to match its memory composition
[
https://issues.apache.org/jira/browse/FLINK-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-14431:
-
Description: This is the umbrella Jira issue collecting all the work on
aligning the TaskManager's metrics overview with the unified memory model for
TaskManagers
([FLIP-49|https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors]).
See
[FLIP-102|https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager]
for further details. (was: h3. Motivation
There are several shortcomings of current (Flink 1.10) Flink TaskManager memory
information show in rest api.
h4. (1) The information from HardwareDescription is difficult to match the
memory compositions of TaskManager in flip-49. As below picture show：
!image-2019-12-19-18-09-05-542.png|width=444,height=389!
* what's the meaning of HardwareDescription.sizeOfJvmHeap.
* the user couldn't get resource config about TaskManager.
h4. (2) There isn't information for managed memory.
* no metric for managed memory.
h4. (3) There isn't information for shuffle memory
* according to TaskManagerMetricsInfo's memorySegmentsTotal(ps: shuffle
segment total size), the user couldn't get shuffle memory.
h4. (4) The metrics in the TaskManager's metrics page do not correspond to the
resource configuration of taskmanager
* It is difficult for users to update taskmanager's resource configuration
based on metrics because users couldn’t find configuration items related to
metrics.
h3. Proposed Changes
h4. Add TaskManageResourceInfo which match the memory compositions
* information from TaskExecutorResourceSpec in flip-49, add it to
TaskExecutorRegistration.
{code:java}
public class TaskManagerResourceInfo {
private final double cpuCores;
private final long frameworkHeap;
private final long frameworkOffHeap;
private final long taskHeap;
private final long taskOffHeap;
private final long shuffleMemory;
private final long managedMemory;
private final long jvmMetaSpace;
private final long jvmOverhead;
private final long totalProcessMemory;
}
{code}
* url: /taskmanagers/:taskmanagerid
* response: add
{code:json}
resource: {
cpuCores: 4,
frameworkHeap: 134217728,
frameworkOffHeap: 134217728,
taskHeap: 181193928,
taskOffHeap: 0,
shuffleMemory: 33554432,
managedMemory: 322122552,
jvmMetaSpace: 134217728,
jvmOverhead: 134217728,
totalProcessMemory: 1073741824
}
{code}
h4. Add shuffle memory metric
* add getTotalMemorySize and in getAvaliableMemorySize NetworkBufferPool
{code:java}
public long getTotalMemorySize() {
return 1L * getTotalNumberOfMemorySegments() * memorySegmentSize;
}
public long getAvaliableMemorySize() {
return 1L * getNumberOfAvailableMemorySegments() * memorySegmentSize;
}{code}
* update NettyShuffleMetricFactory#registerShuffleMetrics
{code:java}
private static final String METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY =
"TotalMemoryCapacity";
private static final String METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY =
"AvaliableMemory";
private static void registerShuffleMetrics(
String groupName,
MetricGroup metricGroup,
NetworkBufferPool networkBufferPool) {
MetricGroup networkGroup = metricGroup.addGroup(groupName);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT,
networkBufferPool::getTotalNumberOfMemorySegments);
networkGroup.>gauge(METRIC_AVAILABLE_MEMORY_SEGMENT,
networkBufferPool::getNumberOfAvailableMemorySegments);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY,
networkBufferPool::getTotalMemorySize);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY,
networkBufferPool::getAvaliableMemorySize);
}
{code}
h4. Add manage memory metric
* add default memory type in MemoryManager
{code:java}
public static final MemoryType DEFAULT_MEMORY_TYPE = MemoryType.OFF_HEAP;
{code}
* add getManagedMemoryTotal in TaskExecutor:
{code:java}
public long getManagedMemoryTotal() {
return this.taskSlotTable.getAllocatedSlots().stream().mapToLong(
slot ->
slot.getMemoryManager().getMemorySizeByType(MemoryManager.DEFAULT_MEMORY_TYPE)
).sum();
}{code}
* add getManagedMemoryUsed in TaskExecutor:
{code:java}
public long getManagedMemoryUsed() {
return this.taskSlotTable.getAllocatedSlots().stream().mapToLong(
slot ->
slot.getMemoryManager().getMemorySizeByType(MemoryManager.DEFAULT_MEMORY_TYPE)
- slot.getMemoryManager().availableMemory(MemoryManager.DEFAULT_MEMORY_TYPE)
).sum();
}{code}
* add instantiateMemoryManagerMetrics in MetricUtils
{code:java}
public static void instantiateMemoryManagerMetrics(MetricGroup
statusMetricGroup, Ta
[GitHub] [flink] rmetzger commented on a change in pull request #13930: [FLINK-19974][python][e2e] Fix the bug that kafka related services are not teardown properly after tests finished
rmetzger commented on a change in pull request #13930:
URL: https://github.com/apache/flink/pull/13930#discussion_r517854585
##
File path: flink-end-to-end-tests/test-scripts/test_pyflink.sh
##
@@ -272,8 +272,6 @@ if [[ "${EXPECTED_MSG[*]}" != "${SORTED_READ_MSG[*]}" ]];
then
exit 1
fi
-stop_cluster
-
Review comment:
I see that there is the `test_clean_up` function which also stops the
cluster. How did this function call prevent the `test_clean_up` function to
work properly?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #13331: [FLINK-19079][table-runtime] Import rowtime deduplicate operator
wuchong commented on a change in pull request #13331:
URL: https://github.com/apache/flink/pull/13331#discussion_r517854922
##
File path:
flink-table/flink-table-runtime-blink/src/test/java/org/apache/flink/table/runtime/operators/deduplicate/RowTimeMiniBatchDeduplicateFunctionTest.java
##
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.deduplicate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.operators.bundle.KeyedMapBundleOperator;
+import
org.apache.flink.table.runtime.operators.bundle.trigger.CountBundleTrigger;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.BinaryRowDataKeySelector;
+import org.apache.flink.table.runtime.util.GenericRowRecordSortComparator;
+import org.apache.flink.table.runtime.util.RowDataHarnessAssertor;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.types.RowKind;
+
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import static
org.apache.flink.table.runtime.util.StreamRecordUtils.insertRecord;
+import static org.apache.flink.table.runtime.util.StreamRecordUtils.record;
+
+/**
+ * Harness tests for {@link RowTimeDeduplicateFunction} and {@link
RowTimeMiniBatchDeduplicateFunction}.
+ */
+@RunWith(Parameterized.class)
+public class RowTimeMiniBatchDeduplicateFunctionTest {
Review comment:
nit: would be better to call `RowTimeDeduplicateFunctionTest`,
`minibatch` is the parameter.
##
File path:
flink-table/flink-table-runtime-blink/src/test/java/org/apache/flink/table/runtime/operators/deduplicate/RowTimeMiniBatchDeduplicateFunctionTest.java
##
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.deduplicate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.streaming.api.operators.KeyedProcessOperator;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.operators.bundle.KeyedMapBundleOperator;
+import
org.apache.flink.table.runtime.operators.bundle.trigger.CountBundleTrigger;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.BinaryRowDataKeySelector;
+import org.apac
[GitHub] [flink] flinkbot commented on pull request #13933: [FLINK-19946][Connectors / HBase]Support sink parallelism configuration for Hbase connector
flinkbot commented on pull request #13933: URL: https://github.com/apache/flink/pull/13933#issuecomment-722212503 ## CI report: * eed3ec1b04aa25fcaafe0ac2003d22e5ff202941 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #13296: [FLINK-18774][format][debezium] Support debezium-avro format
wuchong commented on pull request #13296: URL: https://github.com/apache/flink/pull/13296#issuecomment-722213913 @caozhen1937 , recently we applied some fixes on the `AvroSchemaConverter#convertToSchema`, it doesn't work correctly for duplicate nested field names, see FLINK-19291, FLINK-19779. However, it seems you reverted these changes. Is there any problems with these changes for you? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19912) JSON format fails to serialize map value with null keys
[ https://issues.apache.org/jira/browse/FLINK-19912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226563#comment-17226563 ] Jark Wu commented on FLINK-19912: - Sounds good to me. I will add the solution to the issue description. > JSON format fails to serialize map value with null keys > --- > > Key: FLINK-19912 > URL: https://issues.apache.org/jira/browse/FLINK-19912 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Major > Fix For: 1.12.0 > > > Support ignoreParseErrors to skip error when serializing `RowData` in json > serializationSchema. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13934: [FLINK-19986] Skip license check for scala 2.12 profile
flinkbot commented on pull request #13934: URL: https://github.com/apache/flink/pull/13934#issuecomment-722214429 ## CI report: * a95425118d19a5855166b1ba3ce46a4da804b3ff UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19645) ShuffleCompressionITCase.testDataCompressionForBlockingShuffle is instable
[
https://issues.apache.org/jira/browse/FLINK-19645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19645:
---
Labels: pull-request-available test-stability (was: test-stability)
> ShuffleCompressionITCase.testDataCompressionForBlockingShuffle is instable
> --
>
> Key: FLINK-19645
> URL: https://issues.apache.org/jira/browse/FLINK-19645
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Yingjie Cao
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7631&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code}
> 2020-10-14T21:46:25.3133019Z [ERROR]
> testDataCompressionForBlockingShuffle[useBroadcastPartitioner =
> true](org.apache.flink.test.runtime.ShuffleCompressionITCase) Time elapsed:
> 34.949 s <<< ERROR!
> 2020-10-14T21:46:25.3133688Z org.apache.flink.util.FlinkException: Could not
> close resource.
> 2020-10-14T21:46:25.3134146Z at
> org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:42)
> 2020-10-14T21:46:25.3134716Z at
> org.apache.flink.test.runtime.ShuffleCompressionITCase.executeTest(ShuffleCompressionITCase.java:114)
> 2020-10-14T21:46:25.3135396Z at
> org.apache.flink.test.runtime.ShuffleCompressionITCase.testDataCompressionForBlockingShuffle(ShuffleCompressionITCase.java:89)
> 2020-10-14T21:46:25.3135941Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-14T21:46:25.3136410Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-14T21:46:25.3136959Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-14T21:46:25.3137435Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-14T21:46:25.3137915Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-14T21:46:25.3138459Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-14T21:46:25.3138987Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-14T21:46:25.3139524Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-14T21:46:25.3140017Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-14T21:46:25.3140495Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-14T21:46:25.3141037Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-14T21:46:25.3141520Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-14T21:46:25.3142213Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-14T21:46:25.3142672Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-14T21:46:25.3143134Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-14T21:46:25.3143578Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-14T21:46:25.3144026Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-14T21:46:25.3144437Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-10-14T21:46:25.3144815Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-10-14T21:46:25.3147907Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-14T21:46:25.3148415Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-14T21:46:25.3149058Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-14T21:46:25.3149526Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-14T21:46:25.3149993Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-14T21:46:25.3150429Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-14T21:46:25.3150918Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-14T21:46:25.3151469Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-14T21:46:25.3152045Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-14T21:46:25.3152599Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-10-14T21:46:25.3153163Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoad
[GitHub] [flink] wsry opened a new pull request #13936: [FLINK-19645][tests] Fix akka rpc timeout issue of ShuffleCompressionITCase
wsry opened a new pull request #13936: URL: https://github.com/apache/flink/pull/13936 ## What is the purpose of the change Fix akka rpc timeout issue of ShuffleCompressionITCase by increasing the timeout configuration. ## Brief change log - Fix akka rpc timeout issue of ShuffleCompressionITCase by increasing the timeout configuration. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19912) JSON format fails to serialize map value with null keys
[
https://issues.apache.org/jira/browse/FLINK-19912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226565#comment-17226565
]
Jark Wu commented on FLINK-19912:
-
I saw you first suggest {{json.map-null-key.handling}} which I think is
straightforward. Is there any reason chaning to {{json.map-null-key.mode}}?
> JSON format fails to serialize map value with null keys
> ---
>
> Key: FLINK-19912
> URL: https://issues.apache.org/jira/browse/FLINK-19912
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: hailong wang
>Assignee: hailong wang
>Priority: Major
> Fix For: 1.12.0
>
>
> Support ignoreParseErrors to skip error when serializing `RowData` in json
> serializationSchema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19645) ShuffleCompressionITCase.testDataCompressionForBlockingShuffle is instable
[
https://issues.apache.org/jira/browse/FLINK-19645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226566#comment-17226566
]
Yingjie Cao commented on FLINK-19645:
-
After some investigation, I didn't find any potential bug. The akka rpc timeout
problem were caused by different rpc calls. I think the reason is purely
because this test case is bit heavy. I tried to increase the akka timeout to
60s. The good news is that the problem never reproduce after running the test
case over 2000 times on azure pipeline. If I roll back the timeout, the timeout
issue can be reproduced.
I have opened a PR which just increases the akka timeout. Can anyone kindly
review it? Let's see if it solves the problem.
> ShuffleCompressionITCase.testDataCompressionForBlockingShuffle is instable
> --
>
> Key: FLINK-19645
> URL: https://issues.apache.org/jira/browse/FLINK-19645
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Yingjie Cao
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7631&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code}
> 2020-10-14T21:46:25.3133019Z [ERROR]
> testDataCompressionForBlockingShuffle[useBroadcastPartitioner =
> true](org.apache.flink.test.runtime.ShuffleCompressionITCase) Time elapsed:
> 34.949 s <<< ERROR!
> 2020-10-14T21:46:25.3133688Z org.apache.flink.util.FlinkException: Could not
> close resource.
> 2020-10-14T21:46:25.3134146Z at
> org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:42)
> 2020-10-14T21:46:25.3134716Z at
> org.apache.flink.test.runtime.ShuffleCompressionITCase.executeTest(ShuffleCompressionITCase.java:114)
> 2020-10-14T21:46:25.3135396Z at
> org.apache.flink.test.runtime.ShuffleCompressionITCase.testDataCompressionForBlockingShuffle(ShuffleCompressionITCase.java:89)
> 2020-10-14T21:46:25.3135941Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-14T21:46:25.3136410Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-14T21:46:25.3136959Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-14T21:46:25.3137435Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-14T21:46:25.3137915Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-14T21:46:25.3138459Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-14T21:46:25.3138987Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-14T21:46:25.3139524Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-14T21:46:25.3140017Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-14T21:46:25.3140495Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-14T21:46:25.3141037Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-14T21:46:25.3141520Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-14T21:46:25.3142213Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-14T21:46:25.3142672Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-14T21:46:25.3143134Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-14T21:46:25.3143578Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-14T21:46:25.3144026Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-14T21:46:25.3144437Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-10-14T21:46:25.3144815Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-10-14T21:46:25.3147907Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-14T21:46:25.3148415Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-14T21:46:25.3149058Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-14T21:46:25.3149526Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-14T21:46:25.3149993Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-14T21:46:25.3150429Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-14T21:46:25.3150918Z at
> org.apache.maven.surefire.junit4.JUnit4Provi
[GitHub] [flink] rmetzger commented on pull request #13894: [FLINK-18117][e2e] Dynamically allocate port causing test instabilities
rmetzger commented on pull request #13894: URL: https://github.com/apache/flink/pull/13894#issuecomment-722216319 CI has passed. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13936: [FLINK-19645][tests] Fix akka rpc timeout issue of ShuffleCompressionITCase
flinkbot commented on pull request #13936: URL: https://github.com/apache/flink/pull/13936#issuecomment-722215984 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 55a9612f3705bc53d7159c6f14985872193ce6fb (Thu Nov 05 08:11:28 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #13784: [FLINK-19698][connectors/common] API improvements to the Sources.
becketqin commented on pull request #13784: URL: https://github.com/apache/flink/pull/13784#issuecomment-722216486 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18117) "Kerberized YARN per-job on Docker test" fails with "Could not start hadoop cluster."
[
https://issues.apache.org/jira/browse/FLINK-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger closed FLINK-18117.
--
Resolution: Fixed
Resolved in
https://github.com/apache/flink/commit/358713ba5b1f29fb9befa72c47864d1e49d5e30e
> "Kerberized YARN per-job on Docker test" fails with "Could not start hadoop
> cluster."
> -
>
> Key: FLINK-18117
> URL: https://issues.apache.org/jira/browse/FLINK-18117
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2683&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T06:03:53.2844296Z Creating slave1 ... [32mdone[0m
> 2020-06-04T06:03:53.4981251Z [1BWaiting for hadoop cluster to come up. We
> have been trying for 0 seconds, retrying ...
> 2020-06-04T06:03:58.5980181Z Waiting for hadoop cluster to come up. We have
> been trying for 5 seconds, retrying ...
> 2020-06-04T06:04:03.6997087Z Waiting for hadoop cluster to come up. We have
> been trying for 10 seconds, retrying ...
> 2020-06-04T06:04:08.7910791Z Waiting for hadoop cluster to come up. We have
> been trying for 15 seconds, retrying ...
> 2020-06-04T06:04:13.8921621Z Waiting for hadoop cluster to come up. We have
> been trying for 20 seconds, retrying ...
> 2020-06-04T06:04:18.9648844Z Waiting for hadoop cluster to come up. We have
> been trying for 25 seconds, retrying ...
> 2020-06-04T06:04:24.0381851Z Waiting for hadoop cluster to come up. We have
> been trying for 31 seconds, retrying ...
> 2020-06-04T06:04:29.1220264Z Waiting for hadoop cluster to come up. We have
> been trying for 36 seconds, retrying ...
> 2020-06-04T06:04:34.1882187Z Waiting for hadoop cluster to come up. We have
> been trying for 41 seconds, retrying ...
> 2020-06-04T06:04:39.2784948Z Waiting for hadoop cluster to come up. We have
> been trying for 46 seconds, retrying ...
> 2020-06-04T06:04:44.3843337Z Waiting for hadoop cluster to come up. We have
> been trying for 51 seconds, retrying ...
> 2020-06-04T06:04:49.4703561Z Waiting for hadoop cluster to come up. We have
> been trying for 56 seconds, retrying ...
> 2020-06-04T06:04:54.5463207Z Waiting for hadoop cluster to come up. We have
> been trying for 61 seconds, retrying ...
> 2020-06-04T06:04:59.6650405Z Waiting for hadoop cluster to come up. We have
> been trying for 66 seconds, retrying ...
> 2020-06-04T06:05:04.7500168Z Waiting for hadoop cluster to come up. We have
> been trying for 71 seconds, retrying ...
> 2020-06-04T06:05:09.8177904Z Waiting for hadoop cluster to come up. We have
> been trying for 76 seconds, retrying ...
> 2020-06-04T06:05:14.9751297Z Waiting for hadoop cluster to come up. We have
> been trying for 81 seconds, retrying ...
> 2020-06-04T06:05:20.0336417Z Waiting for hadoop cluster to come up. We have
> been trying for 87 seconds, retrying ...
> 2020-06-04T06:05:25.1627704Z Waiting for hadoop cluster to come up. We have
> been trying for 92 seconds, retrying ...
> 2020-06-04T06:05:30.2583315Z Waiting for hadoop cluster to come up. We have
> been trying for 97 seconds, retrying ...
> 2020-06-04T06:05:35.3283678Z Waiting for hadoop cluster to come up. We have
> been trying for 102 seconds, retrying ...
> 2020-06-04T06:05:40.4184029Z Waiting for hadoop cluster to come up. We have
> been trying for 107 seconds, retrying ...
> 2020-06-04T06:05:45.5388372Z Waiting for hadoop cluster to come up. We have
> been trying for 112 seconds, retrying ...
> 2020-06-04T06:05:50.6155334Z Waiting for hadoop cluster to come up. We have
> been trying for 117 seconds, retrying ...
> 2020-06-04T06:05:55.7225186Z Command: start_hadoop_cluster failed. Retrying...
> 2020-06-04T06:05:55.7237999Z Starting Hadoop cluster
> 2020-06-04T06:05:56.5188293Z kdc is up-to-date
> 2020-06-04T06:05:56.5292716Z master is up-to-date
> 2020-06-04T06:05:56.5301735Z slave2 is up-to-date
> 2020-06-04T06:05:56.5306179Z slave1 is up-to-date
> 2020-06-04T06:05:56.6800566Z Waiting for hadoop cluster to come up. We have
> been trying for 0 seconds, retrying ...
> 2020-06-04T06:06:01.7668291Z Waiting for hadoop cluster to come up. We have
> been trying for 5 seconds, retrying ...
> 2020-06-04T06:06:06.8620265Z Waiting for hadoop cluster to come up. We have
> been trying for 10 seconds, retrying ...
> 2020-06-04T06:06:11.9753596Z Waiting for hadoop cluster to come up. We have
> been trying for 15 seconds, retrying ...
> 2020-06-04T06:06:17.0402846Z Waiting for hadoop clu
[GitHub] [flink] rmetzger merged pull request #13894: [FLINK-18117][e2e] Dynamically allocate port causing test instabilities
rmetzger merged pull request #13894: URL: https://github.com/apache/flink/pull/13894 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19912) JSON format fails to serialize map value with null keys
[ https://issues.apache.org/jira/browse/FLINK-19912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-19912: Description: When JsonRowDataSerializationSchema serializing map data with null keys, it will throw NPE. As discussed following, we propose to introduce format options to solve this problem. was:Support ignoreParseErrors to skip error when serializing `RowData` in json serializationSchema. > JSON format fails to serialize map value with null keys > --- > > Key: FLINK-19912 > URL: https://issues.apache.org/jira/browse/FLINK-19912 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Major > Fix For: 1.12.0 > > > When JsonRowDataSerializationSchema serializing map data with null keys, it > will throw NPE. > As discussed following, we propose to introduce format options to solve this > problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-19661) Implement changes required to resolve FLIP-104
[ https://issues.apache.org/jira/browse/FLINK-19661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias resolved FLINK-19661. -- Resolution: Fixed > Implement changes required to resolve FLIP-104 > -- > > Key: FLINK-19661 > URL: https://issues.apache.org/jira/browse/FLINK-19661 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Web Frontend >Reporter: Matthias >Assignee: Matthias >Priority: Major > Fix For: 1.12.0 > > > This is an umbrella ticket collecting all tasks related to > [FLIP-104|https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19661) Implement changes required to resolve FLIP-104
[ https://issues.apache.org/jira/browse/FLINK-19661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias updated FLINK-19661: - Release Note: FLINK-19661 (FLIP-104) introduced the metrics overview page for the JobManager to align it with the memory model introduced in FLINK-16614 (FLIP-116). Additionally, the goal was to achieve a similar look-and-feel comparing the TaskManager's and the JobManager's metrics overview page. As part of this work, dynamic parameters are now being forwarded to the JobManager. This enabled us to pass the memory configuration for the JobManager through dynamic parameters as it was already done for the TaskManager. The effective memory configuration is now being exposed through the JobManager's configuration page and REST endpoint. > Implement changes required to resolve FLIP-104 > -- > > Key: FLINK-19661 > URL: https://issues.apache.org/jira/browse/FLINK-19661 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Web Frontend >Reporter: Matthias >Assignee: Matthias >Priority: Major > > This is an umbrella ticket collecting all tasks related to > [FLIP-104|https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19912) JSON format fails to serialize map value with null keys
[
https://issues.apache.org/jira/browse/FLINK-19912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-19912:
Description:
When JsonRowDataSerializationSchema serializing map data with null keys, it
will throw NPE.
As discussed following, we propose to introduce format options to solve this
problem.
{{json.map-null-key.mode}} controls the handling mode when encountering null
keys for map data.
- {{FAIL}} (default): throws exception
- {{DROP}}: drop null key entries
- {{LITERAL}}: serialize null keys as string literal which is determined by
{{json.map-null-key.literal}} option.
The default value of {{json.map-null-key.literal}} is "null".
was:
When JsonRowDataSerializationSchema serializing map data with null keys, it
will throw NPE.
As discussed following, we propose to introduce format options to solve this
problem.
> JSON format fails to serialize map value with null keys
> ---
>
> Key: FLINK-19912
> URL: https://issues.apache.org/jira/browse/FLINK-19912
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: hailong wang
>Assignee: hailong wang
>Priority: Major
> Fix For: 1.12.0
>
>
> When JsonRowDataSerializationSchema serializing map data with null keys, it
> will throw NPE.
> As discussed following, we propose to introduce format options to solve this
> problem.
> {{json.map-null-key.mode}} controls the handling mode when encountering null
> keys for map data.
> - {{FAIL}} (default): throws exception
> - {{DROP}}: drop null key entries
> - {{LITERAL}}: serialize null keys as string literal which is determined by
> {{json.map-null-key.literal}} option.
> The default value of {{json.map-null-key.literal}} is "null".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19661) Implement changes required to resolve FLIP-104
[ https://issues.apache.org/jira/browse/FLINK-19661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias updated FLINK-19661: - Fix Version/s: 1.12.0 > Implement changes required to resolve FLIP-104 > -- > > Key: FLINK-19661 > URL: https://issues.apache.org/jira/browse/FLINK-19661 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Web Frontend >Reporter: Matthias >Assignee: Matthias >Priority: Major > Fix For: 1.12.0 > > > This is an umbrella ticket collecting all tasks related to > [FLIP-104|https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] becketqin commented on pull request #13784: [FLINK-19698][connectors/common] API improvements to the Sources.
becketqin commented on pull request #13784: URL: https://github.com/apache/flink/pull/13784#issuecomment-722219407 > Maybe we don't backport the CheckpointListener part, but only the close() method part? Sounds good to me. Personally I don't think backporting this interface change is necessary. If we want to backport the `CheckPointListener` as well. We can just add the `onCheckpointComplete()` and `onCheckpointAborted()` to `SourceReader`/`SplitEnumerator`/`SourceOperator`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #13784: [FLINK-19698][connectors/common] API improvements to the Sources.
becketqin commented on pull request #13784: URL: https://github.com/apache/flink/pull/13784#issuecomment-70211 The CI case failure seems unrelated. I am running the CI test again and will merge the code after that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-14431) Update TaskManager's memory information to match its memory composition
[ https://issues.apache.org/jira/browse/FLINK-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias updated FLINK-14431: - Fix Version/s: 1.12.0 > Update TaskManager's memory information to match its memory composition > --- > > Key: FLINK-14431 > URL: https://issues.apache.org/jira/browse/FLINK-14431 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST, Runtime / Task, Runtime / Web Frontend >Reporter: lining >Assignee: Matthias >Priority: Major > Fix For: 1.12.0 > > Attachments: image-2019-10-17-17-58-50-342.png, > image-2019-10-17-18-01-09-353.png, image-2019-10-17-18-29-53-329.png, > image-2019-10-24-16-19-15-499.png, image-2019-10-24-16-20-23-210.png, > image-2019-10-24-16-22-27-360.png, image-2019-12-19-18-09-05-542.png, > image-2019-12-19-18-27-38-589.png, image-2019-12-19-18-28-01-447.png > > > This is the umbrella Jira issue collecting all the work on aligning the > TaskManager's metrics overview with the unified memory model for TaskManagers > ([FLIP-49|https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors]). > See > [FLIP-102|https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager] > for further details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-14431) Update TaskManager's memory information to match its memory composition
[ https://issues.apache.org/jira/browse/FLINK-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias resolved FLINK-14431. -- Resolution: Fixed > Update TaskManager's memory information to match its memory composition > --- > > Key: FLINK-14431 > URL: https://issues.apache.org/jira/browse/FLINK-14431 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST, Runtime / Task, Runtime / Web Frontend >Reporter: lining >Assignee: Matthias >Priority: Major > Attachments: image-2019-10-17-17-58-50-342.png, > image-2019-10-17-18-01-09-353.png, image-2019-10-17-18-29-53-329.png, > image-2019-10-24-16-19-15-499.png, image-2019-10-24-16-20-23-210.png, > image-2019-10-24-16-22-27-360.png, image-2019-12-19-18-09-05-542.png, > image-2019-12-19-18-27-38-589.png, image-2019-12-19-18-28-01-447.png > > > This is the umbrella Jira issue collecting all the work on aligning the > TaskManager's metrics overview with the unified memory model for TaskManagers > ([FLIP-49|https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors]). > See > [FLIP-102|https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager] > for further details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13280: [FLINK-18070][table-planner-blink] Don't materialize time attribute in SubGraphOptimize
flinkbot edited a comment on pull request #13280: URL: https://github.com/apache/flink/pull/13280#issuecomment-683394013 ## CI report: * Unknown: [CANCELED](TBD) * 3c7c8dba3fcf15fd46a73200c633269f13904723 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9051) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #13897: [FLINK-19939][table-planner-blink] Remove redundant union from multiple input node
godfreyhe commented on a change in pull request #13897:
URL: https://github.com/apache/flink/pull/13897#discussion_r517849708
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/processors/MultipleInputNodeCreationProcessor.java
##
@@ -283,6 +283,102 @@ private void
optimizeMultipleInputGroups(List orderedWrappers)
wrapper.removeFromGroup();
}
}
+
+ // wrappers are checked in topological order from sinks to
sources
+ for (ExecNodeWrapper wrapper : orderedWrappers) {
+ MultipleInputGroup group = wrapper.group;
+ if (group == null) {
+ // we only consider nodes currently in a
multiple input group
+ continue;
+ }
+
+ if (isOutputOfMultipleInputGroup(wrapper) &&
wrapper.execNode instanceof Union) {
+ // optimization 5. this optimization remove
redundant union at the output of a
+ // multiple input, consider the following graph:
+ //
+ // source -> exchange -> agg ---\
+ // source -> exchange -> agg --> union ->
+ // source -> exchange -> join --/
+ // source -> exchange --/
+ //
+ // we'll initially put aggs, the join and the
union into a multiple input, while
+ // the union here is actually redundant.
+ int numberOfUsefulInputs = 0;
+ List uselessBranches = new
ArrayList<>();
+ List>
sameGroupWrappersList = new ArrayList<>();
+
+ // an input branch is useful if it contains a
node with two or more inputs other
+ // than union. we shall keep the union if it
has two or more useful input branches,
+ // as this may benefit source chaining.
consider the following example:
+ //
+ // chainable source -> join -\
+ // / \
+ // chainable source -- join -/
+ for (int i = 0; i < wrapper.inputs.size(); i++)
{
+ ExecNodeWrapper inputWrapper =
wrapper.inputs.get(i);
+ List sameGroupWrappers
= getInputWrappersInSameGroup(inputWrapper, wrapper.group);
+
sameGroupWrappersList.add(sameGroupWrappers);
+ long numberOfValuableNodes =
sameGroupWrappers.stream()
+ .filter(w -> w.inputs.size() >=
2 && !(w.execNode instanceof Union))
+ .count();
+ if (numberOfValuableNodes > 0) {
+ numberOfUsefulInputs++;
+ } else {
+ uselessBranches.add(i);
+ }
+ }
+
+ if (numberOfUsefulInputs < 2) {
+ // remove this union and its useless
branches from multiple input
+ for (int branch : uselessBranches) {
+ List
sameGroupWrappers = sameGroupWrappersList.get(branch);
+ for (ExecNodeWrapper w :
sameGroupWrappers) {
+ if (w.group != null) {
+
w.removeFromGroup();
+ }
+ }
+ }
+ wrapper.removeFromGroup();
+ }
+ }
+ }
+ }
+
+ private List
getInputWrappersInSameGroup(ExecNodeWrapper wrapper, MultipleInputGroup group) {
+ List ret = new ArrayList<>();
+ Queue queue = new LinkedList<>();
+ Set visited = new HashSet<>();
+ queue.add(wrapper);
+ visited.add(wrapper);
+
+ while (!queue.isEmpty()) {
+ ExecNodeWrapper w = que
[GitHub] [flink] wangyang0918 commented on a change in pull request #13864: [FLINK-19543][k8s] Implement RunningJobsRegistry, JobGraphStore based on Kubernetes API
wangyang0918 commented on a change in pull request #13864:
URL: https://github.com/apache/flink/pull/13864#discussion_r517867848
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/zookeeper/ZooKeeperStateHandleStore.java
##
@@ -474,6 +488,8 @@ protected String getLockPath(String rootPath) {
client.create().withMode(CreateMode.EPHEMERAL).forPath(getLockPath(path));
} catch (KeeperException.NodeExistsException ignored) {
// we have already created the lock
+ } catch (KeeperException.NoNodeException ex) {
Review comment:
I will add a comment here so that it is easier to understand.
```
// We could run into this exception because the parent node does not exist
when we are trying to lock.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
