[jira] [Created] (FLINK-20288) Correct documentation about savepoint self-contained
Yun Tang created FLINK-20288: Summary: Correct documentation about savepoint self-contained Key: FLINK-20288 URL: https://issues.apache.org/jira/browse/FLINK-20288 Project: Flink Issue Type: Bug Affects Versions: 1.11.0 Reporter: Yun Tang Assignee: Yun Tang Fix For: 1.12.0, 1.11.4 Savepoint self-contained has been supported while the documentation still remain as not supported, we should fix that description. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20287) Add documentation of how to switch memory allocator in Flink docker image
Yun Tang created FLINK-20287: Summary: Add documentation of how to switch memory allocator in Flink docker image Key: FLINK-20287 URL: https://issues.apache.org/jira/browse/FLINK-20287 Project: Flink Issue Type: Improvement Components: Deployment / Kubernetes, Documentation Reporter: Yun Tang Assignee: Yun Tang Fix For: 1.12.0 Add documentation to tell user how to switch memory allocator in Flink docker image. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Releasing Apache Flink 1.11.3
Thanks for the notice, Zhu. Let's have the discussion on the jira ticket. Thank you~ Xintong Song On Mon, Nov 23, 2020 at 12:10 PM Zhu Zhu wrote: > Hi Xintong, > We just found an issue FLINK-20285 and I think it is a blocker for 1.11.3. > > Thanks, > Zhu > > Gyula Fóra 于2020年11月18日周三 下午9:36写道： > > > Hi All! > > > > I have found the following issue today which might be considered a > blocker > > for this release as well: > > > > https://issues.apache.org/jira/browse/FLINK-20221 > > > > Could someone please quickly provide a second set of eyes and confirm > that > > this is indeed a big problem? :) > > > > Thank you! > > Gyula > > > > On Wed, Nov 18, 2020 at 5:12 AM Xintong Song > > wrote: > > > > > Hi devs, > > > > > > Updates on the progress of preparing the 1.11.3 release. > > > > > > We are approaching the creation of our first release candidate. Thanks > > all > > > for the efforts so far. > > > > > > # Ongoing threads > > > > > >- *FLIP-27 backportings:* 74 out of 102 commits are already > > backported. > > >Stephan and Becket are actively working on this. > > >- *Blockers:* There are 2 remaining blockers, both have been fixed > on > > >the master branch and should be backported along with the FLIP-27 > > > changes. > > > > > > As soon as the above threads are done, we will create our first release > > > candidate. If there's any other issues that you believe should be a > > release > > > blocker, please reach out to either me or Gordon. > > > > > > # JIRA version > > > > > > Version 1.11.4 has been created on JIRA. I've already tagged some of > the > > > tickets with the new fix version. Please feel free to continue with the > > > efforts and merge once they are ready. I'll double check and update the > > fix > > > versions before the release. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song > > > wrote: > > > > > > > Thanks for the notice and fix, Roman. > > > > > > > > Thank you~ > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman < > > > > khachatryan.ro...@gmail.com> wrote: > > > > > > > >> Hi, > > > >> > > > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in > > 1.11.3. > > > >> > > > >> [1] https://issues.apache.org/jira/browse/FLINK-20079 > > > >> > > > >> Regards, > > > >> Roman > > > >> > > > >> > > > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song > > > > >> wrote: > > > >> > > > >> > Thanks for the notice, Dian. > > > >> > > > > >> > Thank you~ > > > >> > > > > >> > Xintong Song > > > >> > > > > >> > > > > >> > > > > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu > > > wrote: > > > >> > > > > >> > > Hi Xintong, > > > >> > > > > > >> > > I want to bring one more issue to your attention [1]. The test > > case > > > >> > > UnalignedCheckpointCompatibilityITCase.test failed several times > > in > > > >> the > > > >> > > last nightly test of release-1.11. We need to figure out if it's > > > just > > > >> an > > > >> > > instable test or caused by recent changes. > > > >> > > > > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065 > > > >> > > > > > >> > > > 在 2020年11月10日，上午9:24，Xintong Song 写道： > > > >> > > > > > > >> > > > Thanks for the replies. > > > >> > > > > > > >> > > > Thank you~ > > > >> > > > > > > >> > > > Xintong Song > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin < > > becket@gmail.com> > > > >> > wrote: > > > >> > > > > > > >> > > >> Hi Xintong, > > > >> > > >> > > > >> > > >> Thanks for driving the release. Just want to sync up on the > > > FLIP-27 > > > >> > > >> backporting. Stephan and I are still trying to backport a > bunch > > > of > > > >> > > patches > > > >> > > >> of Source to 1.11.3. Including: > > > >> > > >> > > > >> > > >> [FLINK-19698][connector/common] Add a close() method to the > > > >> > SplitReader. > > > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return > END_OF_INPUT > > > if > > > >> > > >> SplitReader.fetch throws > > > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job multiple > > > >> times in > > > >> > > >> SourceCoordinator. > > > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final > adjustments. > > > >> > > >> > > > >> > > >> and a few more fixes. > > > >> > > >> > > > >> > > >> We are currently trying to fix them in 1.12 first so it might > > > take > > > >> a > > > >> > > little > > > >> > > >> longer to backport them to 1.11.3. I think it will probably > > take > > > >> us a > > > >> > > few > > > >> > > >> more days to finish the backport. So that would roughly be > the > > > end > > > >> of > > > >> > > this > > > >> > > >> week. > > > >> > > >> > > > >> > > >> Thanks, > > > >> > > >> > > > >> > > >> Jiangjie (Becket) Qin > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann < > > > trohrm...@apache.org >
[jira] [Created] (FLINK-20286) Support streaming source for filesystem SQL connector
Jark Wu created FLINK-20286: --- Summary: Support streaming source for filesystem SQL connector Key: FLINK-20286 URL: https://issues.apache.org/jira/browse/FLINK-20286 Project: Flink Issue Type: New Feature Components: Connectors / FileSystem, Table SQL / Ecosystem Reporter: Jark Wu Currenlty, the filesystem SQL connector only supports bounded source. It would be great to support streaming read just like Hive connector. It should monitor the new added files and read the content of new files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Releasing Apache Flink 1.11.3
Hi Xintong, We just found an issue FLINK-20285 and I think it is a blocker for 1.11.3. Thanks, Zhu Gyula Fóra 于2020年11月18日周三 下午9:36写道： > Hi All! > > I have found the following issue today which might be considered a blocker > for this release as well: > > https://issues.apache.org/jira/browse/FLINK-20221 > > Could someone please quickly provide a second set of eyes and confirm that > this is indeed a big problem? :) > > Thank you! > Gyula > > On Wed, Nov 18, 2020 at 5:12 AM Xintong Song > wrote: > > > Hi devs, > > > > Updates on the progress of preparing the 1.11.3 release. > > > > We are approaching the creation of our first release candidate. Thanks > all > > for the efforts so far. > > > > # Ongoing threads > > > >- *FLIP-27 backportings:* 74 out of 102 commits are already > backported. > >Stephan and Becket are actively working on this. > >- *Blockers:* There are 2 remaining blockers, both have been fixed on > >the master branch and should be backported along with the FLIP-27 > > changes. > > > > As soon as the above threads are done, we will create our first release > > candidate. If there's any other issues that you believe should be a > release > > blocker, please reach out to either me or Gordon. > > > > # JIRA version > > > > Version 1.11.4 has been created on JIRA. I've already tagged some of the > > tickets with the new fix version. Please feel free to continue with the > > efforts and merge once they are ready. I'll double check and update the > fix > > versions before the release. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song > > wrote: > > > > > Thanks for the notice and fix, Roman. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman < > > > khachatryan.ro...@gmail.com> wrote: > > > > > >> Hi, > > >> > > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in > 1.11.3. > > >> > > >> [1] https://issues.apache.org/jira/browse/FLINK-20079 > > >> > > >> Regards, > > >> Roman > > >> > > >> > > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song > > >> wrote: > > >> > > >> > Thanks for the notice, Dian. > > >> > > > >> > Thank you~ > > >> > > > >> > Xintong Song > > >> > > > >> > > > >> > > > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu > > wrote: > > >> > > > >> > > Hi Xintong, > > >> > > > > >> > > I want to bring one more issue to your attention [1]. The test > case > > >> > > UnalignedCheckpointCompatibilityITCase.test failed several times > in > > >> the > > >> > > last nightly test of release-1.11. We need to figure out if it's > > just > > >> an > > >> > > instable test or caused by recent changes. > > >> > > > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065 > > >> > > > > >> > > > 在 2020年11月10日，上午9:24，Xintong Song 写道： > > >> > > > > > >> > > > Thanks for the replies. > > >> > > > > > >> > > > Thank you~ > > >> > > > > > >> > > > Xintong Song > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin < > becket@gmail.com> > > >> > wrote: > > >> > > > > > >> > > >> Hi Xintong, > > >> > > >> > > >> > > >> Thanks for driving the release. Just want to sync up on the > > FLIP-27 > > >> > > >> backporting. Stephan and I are still trying to backport a bunch > > of > > >> > > patches > > >> > > >> of Source to 1.11.3. Including: > > >> > > >> > > >> > > >> [FLINK-19698][connector/common] Add a close() method to the > > >> > SplitReader. > > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT > > if > > >> > > >> SplitReader.fetch throws > > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job multiple > > >> times in > > >> > > >> SourceCoordinator. > > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final adjustments. > > >> > > >> > > >> > > >> and a few more fixes. > > >> > > >> > > >> > > >> We are currently trying to fix them in 1.12 first so it might > > take > > >> a > > >> > > little > > >> > > >> longer to backport them to 1.11.3. I think it will probably > take > > >> us a > > >> > > few > > >> > > >> more days to finish the backport. So that would roughly be the > > end > > >> of > > >> > > this > > >> > > >> week. > > >> > > >> > > >> > > >> Thanks, > > >> > > >> > > >> > > >> Jiangjie (Becket) Qin > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann < > > trohrm...@apache.org > > >> > > > >> > > wrote: > > >> > > >> > > >> > > >>> Yes, I've downgraded FLINK-19816 to critical. > > >> > > >>> > > >> > > >>> Cheers, > > >> > > >>> Till > > >> > > >>> > > >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song < > > >> tonysong...@gmail.com> > > >> > > >>> wrote: > > >> > > >>> > > >> > > Thanks for the notice, Till. > > >> > > > > >> > > I just checked and found FLINK-20033 is already fixed. Shall > we > > >> also > > >> > > downgrade FLINK-19816 to `Critical`?
[jira] [Created] (FLINK-20285) LazyFromSourcesSchedulingStrategy is possible to schedule non-CREATED vertices
Zhu Zhu created FLINK-20285:
---
Summary: LazyFromSourcesSchedulingStrategy is possible to schedule
non-CREATED vertices
Key: FLINK-20285
URL: https://issues.apache.org/jira/browse/FLINK-20285
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Zhu Zhu
Fix For: 1.12.0, 1.11.3
LazyFromSourcesSchedulingStrategy is possible to schedule vertices which are
not in CREATED state. This will lead result in unexpected check failure and
result in fatal error[1].
The reason is that the status of a vertex to schedule was changed in
LazyFromSourcesSchedulingStrategy#allocateSlotsAndDeployExecutionVertices()
during the invocation of schedulerOperations.allocateSlotsAndDeploy(...) on
other vertices.
e.g. ev1 and ev2 are in the same pipelined region and are restarted one by one
in the scheduling loop in
LazyFromSourcesSchedulingStrategy#allocateSlotsAndDeployExecutionVertices().
They are all CREATED at the moment. ev1 is scheduled first but it immediately
fails due to some slot allocation error and ev2 will be canceled as a result.
So when ev2 is scheduled, its state would be CANCELED and the state check
failed.
[1]
{code:java}
2020-11-19 13:34:17,231 ERROR
org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread
'flink-akka.actor.default-dispatcher-15' produced an uncaught exception.
Stopping the process...
java.util.concurrent.CompletionException: java.lang.IllegalStateException:
expected vertex aafcbb93173905cec9672e46932d7790_3 to be in CREATED state, was:
CANCELED
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
~[?:1.8.0_222]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
~[?:1.8.0_222]
at
java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:708)
~[?:1.8.0_222]
at
java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:687)
~[?:1.8.0_222]
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
~[?:1.8.0_222]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
~[flink-dist_2.11-1.11.2.jar:1.11.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
~[flink-dist_2.11-1.11.2.jar:1.11.2]
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
~[flink-dist_2.11-1.11.2.jar:1.11.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
~[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.11.2.jar:1.11.2]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.11.2.jar:1.11.2]
Caused by: java.lang.IllegalStateException: expected vertex
aafcbb93173905cec9672e46932d7790_3 to be in CREATED state, was: CANCELED
at
[jira] [Created] (FLINK-20284) Error happens in TaskExecutor when closing JobMaster connection if there was a python UDF
Zhu Zhu created FLINK-20284:
---
Summary: Error happens in TaskExecutor when closing JobMaster
connection if there was a python UDF
Key: FLINK-20284
URL: https://issues.apache.org/jira/browse/FLINK-20284
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.12.0
Reporter: Zhu Zhu
Fix For: 1.12.0
When a TaskExecutor successfully finished running a python UDF task and
disconnecting from JobMaster, errors below will happen. This error, however,
seems not affect job execution at the moment.
{code:java}
2020-11-20 17:05:21,932 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - 1 Beam Fn
Logging clients still connected during shutdown.
2020-11-20 17:05:21,938 WARN
org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer[] - Hanged up for
unknown endpoint.
2020-11-20 17:05:22,126 INFO org.apache.flink.runtime.taskmanager.Task
[] - Source: Custom Source -> select: (f0) -> select: (add_one(f0)
AS a) -> to: Tuple2 -> Sink: Streaming select table sink (1/1)#0
(b0c2104dd8f87bb1caf0c83586c22a51) switched from RUNNING to FINISHED.
2020-11-20 17:05:22,126 INFO org.apache.flink.runtime.taskmanager.Task
[] - Freeing task resources for Source: Custom Source -> select:
(f0) -> select: (add_one(f0) AS a) -> to: Tuple2 -> Sink: Streaming select
table sink (1/1)#0 (b0c2104dd8f87bb1caf0c83586c22a51).
2020-11-20 17:05:22,128 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Un-registering task and sending final execution state FINISHED to JobManager
for task Source: Custom Source -> select: (f0) -> select: (add_one(f0) AS a) ->
to: Tuple2 -> Sink: Streaming select table sink (1/1)#0
b0c2104dd8f87bb1caf0c83586c22a51.
2020-11-20 17:05:22,156 INFO
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Free slot
TaskSlot(index:0, state:ACTIVE, resource profile:
ResourceProfile{cpuCores=1., taskHeapMemory=384.000mb
(402653174 bytes), taskOffHeapMemory=0 bytes, managedMemory=512.000mb
(536870920 bytes), networkMemory=128.000mb (134217730 bytes)}, allocationId:
b67c3307dcf93757adfb4f0f9f7b8c7b, jobId: d05f32162f38ec3ec813c4621bc106d9).
2020-11-20 17:05:22,157 INFO
org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Remove job
d05f32162f38ec3ec813c4621bc106d9 from job leader monitoring.
2020-11-20 17:05:22,157 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close
JobManager connection for job d05f32162f38ec3ec813c4621bc106d9.
2020-11-20 17:05:23,064 ERROR
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.DefaultPromise.rejectedExecution
[] - Failed to submit a listener notification task. Event loop shut down?
java.lang.NoClassDefFoundError:
org/apache/beam/vendor/grpc/v1p26p0/io/netty/util/concurrent/GlobalEventExecutor$2
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:227)
~[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:215)
~[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841)
[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498)
[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96)
[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1089)
[blob_p-bd7a5d615512eb8a2e856e7c1630a0c22fca7cf3-ff27946fda7e2b8cb24ea56d505b689e:1.12-SNAPSHOT]
at
org.apache.beam.vendor.grpc.v1p26p0.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[jira] [Created] (FLINK-20283) Make invalid managed memory fraction errors of python udf more user friendly
Zhu Zhu created FLINK-20283: --- Summary: Make invalid managed memory fraction errors of python udf more user friendly Key: FLINK-20283 URL: https://issues.apache.org/jira/browse/FLINK-20283 Project: Flink Issue Type: Improvement Components: API / Python Affects Versions: 1.12.0 Reporter: Zhu Zhu Fix For: 1.12.0 When managed memory is required for python udf but its "taskmanager.memory.managed.consumer-weights" is set to 0, error will happen but the message is hard to understand for users, see [1]. I think we should expose the invalid fraction error to users in this case and guide users to properly configure "taskmanager.memory.managed.consumer-weights". [1] org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78) at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:224) at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:217) at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:208) at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:534) at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89) at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:419) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:286) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:201) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:154) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at akka.actor.ActorCell.invoke(ActorCell.scala:561) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at akka.dispatch.Mailbox.run(Mailbox.scala:225) at akka.dispatch.Mailbox.exec(Mailbox.scala:235) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.IllegalArgumentException at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:126) at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:254) at org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.open(AbstractPythonFunctionOperator.java:121) at org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator.open(AbstractStatelessFunctionOperator.java:134) at org.apache.flink.table.runtime.operators.python.scalar.AbstractPythonScalarFunctionOperator.open(AbstractPythonScalarFunctionOperator.java:94) at org.apache.flink.table.runtime.operators.python.scalar.AbstractRowPythonScalarFunctionOperator.open(AbstractRowPythonScalarFunctionOperator.java:67) at org.apache.flink.table.runtime.operators.python.scalar.PythonScalarFunctionOperator.open(PythonScalarFunctionOperator.java:64) at
[jira] [Created] (FLINK-20282) Make invalid managed memory fraction errors more advisory in MemoryManager
Zhu Zhu created FLINK-20282: --- Summary: Make invalid managed memory fraction errors more advisory in MemoryManager Key: FLINK-20282 URL: https://issues.apache.org/jira/browse/FLINK-20282 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Affects Versions: 1.12.0 Reporter: Zhu Zhu Fix For: 1.12.0 The invalid managed memory fraction errors[1] reported from MemoryManager are not advisory for users to solve the problem. This error happens when managed memory is required for a use case but its weight is 0. I think it would be better to enrich the error message to guide users to properly configure "taskmanager.memory.managed.consumer-weights". [1] "Caused by: java.lang.IllegalArgumentException: The fraction of memory to allocate must within (0, 1], was: 0.0" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20281) Window aggregation supports changelog stream input
Jark Wu created FLINK-20281: --- Summary: Window aggregation supports changelog stream input Key: FLINK-20281 URL: https://issues.apache.org/jira/browse/FLINK-20281 Project: Flink Issue Type: New Feature Components: Table SQL / Planner, Table SQL / Runtime Reporter: Jark Wu Currently, window aggregation doesn't support to consume a changelog stream. This makes it impossible to do a window aggregation on changelog sources (e.g. Kafka with Debezium format, or upsert-kafka, or mysql-cdc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20280) Support batch mode for Python DataStream API
Dian Fu created FLINK-20280: --- Summary: Support batch mode for Python DataStream API Key: FLINK-20280 URL: https://issues.apache.org/jira/browse/FLINK-20280 Project: Flink Issue Type: New Feature Components: API / Python Reporter: Dian Fu Fix For: 1.13.0 Currently, it still doesn't support batch mode for the Python DataStream API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20279) Elasticsearch6DynamicSinkITCase failed caused by pulling docker image error
Leonard Xu created FLINK-20279: -- Summary: Elasticsearch6DynamicSinkITCase failed caused by pulling docker image error Key: FLINK-20279 URL: https://issues.apache.org/jira/browse/FLINK-20279 Project: Flink Issue Type: Task Affects Versions: 1.12.0 Reporter: Leonard Xu I found this Error in a document PR build, please see: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9875=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20 This should be related with FLINK-17159 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20278) Throw a meaningful exception if the Python DataStream API job executes in batch mode
Dian Fu created FLINK-20278: --- Summary: Throw a meaningful exception if the Python DataStream API job executes in batch mode Key: FLINK-20278 URL: https://issues.apache.org/jira/browse/FLINK-20278 Project: Flink Issue Type: Improvement Components: API / Python Affects Versions: 1.12.0 Reporter: Dian Fu Fix For: 1.12.0 Currently, the Python DataStream job still doesn't support batch mode. We should thrown a meaningful exception if it runs in batch mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20277) flink-1.11.2 ContinuousFileMonitoringFunction cannot restore from failure
谢波 created FLINK-20277: -- Summary: flink-1.11.2 ContinuousFileMonitoringFunction cannot restore from failure Key: FLINK-20277 URL: https://issues.apache.org/jira/browse/FLINK-20277 Project: Flink Issue Type: Bug Components: Table SQL / Ecosystem Affects Versions: 1.11.2 Reporter: 谢波 Fix For: 1.11.3 流式消费Hive表，出现失败时，任务无法正常恢复，一直重启。 一直报错：The ContinuousFileMonitoringFunction has already restored from a previous Flink version. java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/rawdata/db/bw_hana/sapecc/hepecc_ekko_cut at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1270) ~[hadoop-hdfs-2.6.0-cdh5.16.2.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262) ~[hadoop-hdfs-2.6.0-cdh5.16.2.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.6.0-cdh5.16.2.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262) ~[hadoop-hdfs-2.6.0-cdh5.16.2.jar:?] at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:85) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:588) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.getInputSplitsSortedByModTime(ContinuousFileMonit oringFunction.java:279) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.monitorDirAndForwardSplits(ContinuousFileMonitori ngFunction.java:251) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.run(ContinuousFileMonitoringFunction.java:215) ~[ flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) ~[flink-dist_2 .11-1.11.2.jar:1.11.2] 2020-11-23 05:00:33,313 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Split Reader: HiveFileMonitoringFunction -> S ink: Sink(table=[default_catalog.default_database.kafka_hepecc_ekko_cut_json], fields=[mandt, ebeln, bukrs, bstyp, bsart, bsakz, loekz, statu , aedat, ernam, pincr, lponr, lifnr, spras, zterm, zbd1t, zbd2t, zbd3t, zbd1p, zbd2p, ekorg, ekgrp, waers, wkurs, kufix, bedat, kdatb, kdate, bwbdt, angdt, bnddt, gwldt, ausnr, angnr, ihran, ihrez, verkf, telf1, llief, kunnr, konnr, abgru, autlf, weakt, reswk, lblif, inco1, inco2, ktwrt, submi, knumv, kalsm, stafo, lifre, exnum, unsez, logsy, upinc, stako, frggr, frgsx, frgke, frgzu, frgrl, lands, lphis, adrnr, stceg_l, stceg, absgr, addnr, kornr, memory, procstat, rlwrt, revno, scmproc, reason_code, memorytype, rettp, retpc, dptyp, dppct, dpamt, dpdat, msr_ id, hierarchy_exists, threshold_exists, legal_contract, description, release_date, force_id, force_cnt, reloc_id, reloc_seq_id, source_logsys , auflg, yxcort, ysyb, ypsfs, yxqlx, yjplx, ylszj, yxdry, yxdrymc, ylbid1, yqybm, ysxpt_order, yy_write_if, fanpfg, yresfg, yrcofg, yretxt, y ...skipping... java.lang.IllegalArgumentException: The ContinuousFileMonitoringFunction has already restored from a previous Flink version. at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.initializeState(ContinuousFileMonitoringFunction.java:176) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
[jira] [Created] (FLINK-20276) Transparent DeCompression of streams missing on new File Source
Stephan Ewen created FLINK-20276:
Summary: Transparent DeCompression of streams missing on new File
Source
Key: FLINK-20276
URL: https://issues.apache.org/jira/browse/FLINK-20276
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.12.0
The existing {{FileInputFormat}} applies decompression (gzip, xy, ...)
automatically on the file input stream, based on the file extension.
We need to add similar functionality for the {{StreamRecordFormat}} of the new
FileSource to be on par with this functionality.
This can be easily applied in the {{StreamFormatAdapter}} when opening the file
stream.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20275) The path delimiter for add_jars and add_classpaths in Python StreamExecutionEnvironment should be ;
Shuiqiang Chen created FLINK-20275: -- Summary: The path delimiter for add_jars and add_classpaths in Python StreamExecutionEnvironment should be ; Key: FLINK-20275 URL: https://issues.apache.org/jira/browse/FLINK-20275 Project: Flink Issue Type: Bug Reporter: Shuiqiang Chen Fix For: 1.12.0 Currently, the path delimiter for add_jars and add_classpaths in Python StreamExecutionEnvironment is ",", this would cause the rest client fail to upload the specified jars and stuck forever without errors. It should be ";" instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20274) Translate page 'testing' into Chinese
Flora Tao created FLINK-20274: - Summary: Translate page 'testing' into Chinese Key: FLINK-20274 URL: https://issues.apache.org/jira/browse/FLINK-20274 Project: Flink Issue Type: Improvement Components: chinese-translation Affects Versions: 1.11.2, 1.12.0 Reporter: Flora Tao the page url is [testing|https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/stream/testing.html] The doc located in "flink/docs/dev/stream/testing.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20273) Fix Table api Kafka connector Sink Partitioner Document Error
Shengkai Fang created FLINK-20273:
-
Summary: Fix Table api Kafka connector Sink Partitioner Document
Error
Key: FLINK-20273
URL: https://issues.apache.org/jira/browse/FLINK-20273
Project: Flink
Issue Type: Bug
Components: Documentation, Table SQL / API
Affects Versions: 1.12.0
Reporter: Shengkai Fang
The
[doc|https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/kafka.html#sink-partitioning]
tells us that the kafka sink uses fixed partitioner by default. However, in my
local test, the it uses sticky partitioner to get the record partition id if
key is not set.
You can add the test in the {{KafkaTableITCase}}, the code follows
{code:java}
public void testKafkaSourceSinkWithDefaultPartitioner() throws Exception {
if (isLegacyConnector) {
return;
}
// we always use a different topic name for each parameterized
topic,
// in order to make sure the topic can be created.
final String topic = "key_full_value_topic_" + format;
createTestTopic(topic, 3, 1);
// -- Produce an event time stream into Kafka
---
String groupId = standardProps.getProperty("group.id");
String bootstraps =
standardProps.getProperty("bootstrap.servers");
// compared to the partial value test we cannot support both
k_user_id and user_id in a full
// value due to duplicate names after key prefix stripping,
// fields are reordered on purpose,
// fields for keys and values are overlapping
final String createSourceTable = String.format(
"CREATE TABLE kafkaSource (\n"
+ " `user_id` BIGINT,\n"
+ " `name` STRING,\n"
+ " `partition` INT METADATA"
+ ") WITH (\n"
+ " 'connector' = 'kafka',\n"
+ " 'topic' = '%s',\n"
+ "
'properties.bootstrap.servers' = '%s',\n"
+ " 'properties.group.id' =
'%s',\n"
+ " 'scan.startup.mode' =
'earliest-offset',\n"
+ " 'format' = '%s'\n"
+ ")",
topic,
bootstraps,
groupId,
format);
final String createSinkTable = String.format(
"CREATE TABLE kafkaSink (\n"
+ " `user_id` BIGINT,\n"
+ " `name` STRING\n"
+ ") WITH (\n"
+ " 'connector' = 'kafka',\n"
+ " 'topic' = '%s',\n"
+ "
'properties.bootstrap.servers' = '%s',\n"
+ " 'properties.group.id' =
'%s',\n"
+ " 'scan.startup.mode' =
'earliest-offset',\n"
+ " 'format' = '%s'\n"
+ ")",
topic,
bootstraps,
groupId,
format);
tEnv.executeSql(createSourceTable);
tEnv.executeSql(createSinkTable);
String initialValues = "INSERT INTO kafkaSink\n"
+
"VALUES\n"
+ " (1,
'name 1'),\n"
+ " (2,
'name 2'),\n"
+ " (3,
'name 3')";
tEnv.executeSql(initialValues).await();
initialValues = "INSERT INTO kafkaSink\n"
+ "VALUES\n"
+ " (4, 'name 4'),\n"
+ " (5, 'name 5'),\n"
+ " (6, 'name 6')";
tEnv.executeSql(initialValues).await();
initialValues = "INSERT INTO kafkaSink\n"
+
