[jira] [Created] (FLINK-22779) KafkaChangelogTableITCase.testKafkaDebeziumChangelogSource fail due to ConcurrentModificationException

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22779:
-

 Summary: 
KafkaChangelogTableITCase.testKafkaDebeziumChangelogSource fail due to 
ConcurrentModificationException
 Key: FLINK-22779
 URL: https://issues.apache.org/jira/browse/FLINK-22779
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18330=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6608





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

The partition tracker should support remote shuffle properly

[XING JIN]

Hi devs ~
Recently our team designed and started to build Flink remote shuffle
service based on 'pluggable shuffle service framework'[1] for batch
processing jobs. We found some potential enhancements could be made on
'pluggable shuffle service' and created an umbrella JIRA[2]. I raise this
DISCUSSION and want to hear broader feedback / comments on one ticket [3]
-- "The partition tracker should support remote shuffle properly".

In current Flink, data partition is bound with the ResourceID of TM in
Execution#startTrackingPartitions and JM partition tracker will stop
tracking corresponding partitions when a TM
disconnects(JobMaster#disconnectTaskManager), i.e. the lifecycle of shuffle
data is bound with computing resource (TM). It works fine for internal
shuffle service, but doesn't for remote shuffle service. Note that shuffle
data is accommodated on remote, the lifecycle of a completed partition is
capable to be decoupled with TM, i.e. TM is totally fine to be released
when no computing task is on it and further shuffle reading requests could
be directed to remote shuffle cluster. In addition, when a TM is lost, its
completed data partitions on remote shuffle cluster could avoid reproducing.

The issue mentioned above is because Flink JobMasterPartitionTracker mixed
up partition's locationID (where the partition is located) and tmID (which
TM the partition is produced from). In TM internal shuffle, partition's
locationID is the same with tmID, but it is not in remote shuffle;
JobMasterPartitionTracker as an independent component should be able to
differentiate locationID and tmID of a partition, thus to handle the
lifecycle of a partition properly;

We propose that JobMasterPartitionTracker indexes partitions with both
locationID and tmID. The process of registration and unregistration will be
like below:

A. Partition Registration
- Execution#registerProducedPartitions registers partition to ShuffleMaster
and gets a ShuffleDescriptor. Current
ShuffleDescriptor#storesLocalResourcesOn returns the location of the
producing TM ONLY IF the partition occupies local resources there. We
propose to change this method a proper name and always return the
locationID of the partition. It might be as below:
ResourceID getLocationID();
- Execution#registerProducePartitions then registers partition to
JMPartitionTracker with tmID (ResourceID of TaskManager from
TaskManagerLocation) and the locationID (acquired in above step).
JobMasterPartitionTracker will indexes a partition with both tmID and
locationID;

B. Invokes from JM and ShuffleMaster
JobMasterPartitionTracker listens invokes from both JM and ShuffleMaster.
- When JMPartitionTracker hears from JobMaster#disconnectTaskManager that a
TM disconnects, it will check whether the disconnected tmID equals a
certain locationID of a partition. If so, tracking of the corresponding
partition will be stopped.
- When JobMasterPartitionTracker hears from ShuffleMaster that a data
location gets lost, it will unregister corresponding partitions by
locationID;

C. Partition Unregistration
When unregister a partition, JobMasterPartitionTracker removes the
corresponding indexes to tmID and locationID firstly, and then release the
partition according to shuffle service types:
- If the locationID equals to the tmID, it indicates the partition is
accommodated by TM internal shuffle service, JMPartitionTracker will invoke
TaskExecutorGateway for the release;
- If the locationID doesn't equal to tmID, it indicates the partition is
accommodated by external shuffle service, JMPartitionTracker will invoke
ShuffleMaster for the release;

With the above change, JobMasterPartitionTracker can adapt with customized
shuffle service properly for partition's lifecycle.

Looking forward to inputs on this ~~

Best,
Jin


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-31%3A+Pluggable+Shuffle+Service
[2] https://issues.apache.org/jira/browse/FLINK-22672
[3] https://issues.apache.org/jira/browse/FLINK-22676

[jira] [Created] (FLINK-22778) Upgrade to JUnit 4.13

[Thomas Weise (Jira)]

Thomas Weise created FLINK-22778:


 Summary: Upgrade to JUnit 4.13
 Key: FLINK-22778
 URL: https://issues.apache.org/jira/browse/FLINK-22778
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Affects Versions: 1.13.0
Reporter: Thomas Weise
Assignee: Thomas Weise


The latest JUnit4 release has useful improvements and fixes: 
https://github.com/junit-team/junit4/blob/HEAD/doc/ReleaseNotes4.13.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22777) Recover lost parts of Datastream API Try Flink example

[Stefan Gloutnikov (Jira)]

Stefan Gloutnikov created FLINK-22777:
-

 Summary: Recover lost parts of Datastream API Try Flink example 
 Key: FLINK-22777
 URL: https://issues.apache.org/jira/browse/FLINK-22777
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.14.0
Reporter: Stefan Gloutnikov


During the migration of the docs from Jekyll to Hugo (FLINK-21193) in 1.13 
large parts of the fraud detection Datastream API example in the "Try Flink" 
section were lost. Restore it for future doc versions. 

 
1.12 - 
[https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html]
1.13 - 
[https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/try-flink/datastream/]
 
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22776) Delete casting to byte[]

[Ada Wong (Jira)]

Ada Wong created FLINK-22776:


 Summary: Delete casting to byte[]
 Key: FLINK-22776
 URL: https://issues.apache.org/jira/browse/FLINK-22776
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Affects Versions: 1.13.0
Reporter: Ada Wong
 Attachments: image-2021-05-26-10-38-04-578.png

Casting to 'byte[]' is redundant. Could we delete it?

 !image-2021-05-26-10-38-04-578.png|thumbnail! 





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22775) CassandraConnectorITCase.testCassandraTableSink Fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-22775:
-

 Summary: CassandraConnectorITCase.testCassandraTableSink Fail
 Key: FLINK-22775
 URL: https://issues.apache.org/jira/browse/FLINK-22775
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Cassandra
Affects Versions: 1.14.0
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18328=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=bfbc6239-57a0-5db0-63f3-41551b4f7d51=14105


{code:java}
 2021-05-25T23:03:44.0756266Z May 25 23:03:44 [ERROR] 
testCassandraTableSink(org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase)
  Time elapsed: 13.673 s  <<< ERROR!
2021-05-25T23:03:44.0757635Z May 25 23:03:44 
java.util.concurrent.ExecutionException: 
org.apache.flink.table.api.TableException: Failed to wait job finish
2021-05-25T23:03:44.0760262Z May 25 23:03:44at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2021-05-25T23:03:44.0761504Z May 25 23:03:44at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2021-05-25T23:03:44.0762906Z May 25 23:03:44at 
org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:129)
2021-05-25T23:03:44.0763878Z May 25 23:03:44at 
org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:92)
2021-05-25T23:03:44.0764918Z May 25 23:03:44at 
org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase.testCassandraTableSink(CassandraConnectorITCase.java:520)
2021-05-25T23:03:44.0768225Z May 25 23:03:44at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-05-25T23:03:44.0769100Z May 25 23:03:44at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-05-25T23:03:44.0769917Z May 25 23:03:44at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-05-25T23:03:44.0770645Z May 25 23:03:44at 
java.lang.reflect.Method.invoke(Method.java:498)
2021-05-25T23:03:44.0771387Z May 25 23:03:44at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-05-25T23:03:44.0772228Z May 25 23:03:44at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-05-25T23:03:44.0773541Z May 25 23:03:44at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-05-25T23:03:44.0774367Z May 25 23:03:44at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-05-25T23:03:44.0775246Z May 25 23:03:44at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-25T23:03:44.0776088Z May 25 23:03:44at 
org.apache.flink.testutils.junit.RetryRule$RetryOnExceptionStatement.evaluate(RetryRule.java:192)
2021-05-25T23:03:44.0776946Z May 25 23:03:44at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-05-25T23:03:44.0777685Z May 25 23:03:44at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-05-25T23:03:44.0778447Z May 25 23:03:44at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-25T23:03:44.0779110Z May 25 23:03:44at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-05-25T23:03:44.0779893Z May 25 23:03:44at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-05-25T23:03:44.0780744Z May 25 23:03:44at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-05-25T23:03:44.0781493Z May 25 23:03:44at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-05-25T23:03:44.0782154Z May 25 23:03:44at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-05-25T23:03:44.0782899Z May 25 23:03:44at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-05-25T23:03:44.0783576Z May 25 23:03:44at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-05-25T23:03:44.0784312Z May 25 23:03:44at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-05-25T23:03:44.0785020Z May 25 23:03:44at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-05-25T23:03:44.0785815Z May 25 23:03:44at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2021-05-25T23:03:44.0786619Z May 25 23:03:44at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-05-25T23:03:44.0787343Z May 25 23:03:44at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-05-25T23:03:44.0788202Z May 25 23:03:44at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-05-25T23:03:44.0789018Z May 25 23:03:44at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)

Re: [DISCUSS] Watermark propagation with Sink API

[Arvid Heise]

Hi Eron,

I think the FLIP is crisp and mostly good to go. Some smaller
things/questions:

   1. SinkFunction#writeWatermark could be named
   SinkFunction#invokeWatermark or invokeOnWatermark to keep it symmetric.
   2. We could add the context parameter to both. For SinkWriter#Context,
   we currently do not gain much. SinkFunction#Context also exposes processing
   time, which may or may not be handy and is currently mostly used for
   StreamingFileSink bucket policies. We may add that processing time flag
   also to SinkWriter#Context in the future.
   3. Alternatively, we could also add a different context parameter just
   to keep the API stable while allowing additional information to be passed
   in the future.
   4. Would we run into any compatibility issue if we use Flink 1.13 source
   in Flink 1.14 (with this FLIP) or vice versa?
   5. What happens with sinks that use the new methods in applications that
   do not have watermarks (batch mode, processing time)? Does this also work
   with ingestion time sufficiently?
   6. How do exactly once sinks deal with written watermarks in case of
   failure? I guess it's the same as normal records. (Either rollback of
   transaction or deduplication on resumption)

Best,

Arvid

On Tue, May 25, 2021 at 6:44 PM Eron Wright 
wrote:

> Does anyone have further comment on FLIP-167?
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API
>
> Thanks,
> Eron
>
>
> On Thu, May 20, 2021 at 5:02 PM Eron Wright 
> wrote:
>
> > Filed FLIP-167: Watermarks for Sink API:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API
> >
> > I'd like to call a vote next week, is that reasonable?
> >
> >
> > On Wed, May 19, 2021 at 6:28 PM Zhou, Brian  wrote:
> >
> >> Hi Arvid and Eron,
> >>
> >> Thanks for the discussion and I read through Eron's pull request and I
> >> think this can benefit Pravega Flink connector as well.
> >>
> >> Here is some background. Pravega had the watermark concept through the
> >> event stream since two years ago, and here is a blog introduction[1] for
> >> Pravega watermark.
> >> Pravega Flink connector also had this watermark integration last year
> >> that we wanted to propagate the Flink watermark to Pravega in the
> >> SinkFunction, and at that time we just used the existing Flink API that
> we
> >> keep the last watermark in memory and check if watermark changes for
> each
> >> event[2] which is not efficient. With such new interface, we can also
> >> manage the watermark propagation much more easily.
> >>
> >> [1] https://pravega.io/blog/2019/11/08/pravega-watermarking-support/
> >> [2]
> >>
> https://github.com/pravega/flink-connectors/blob/master/src/main/java/io/pravega/connectors/flink/FlinkPravegaWriter.java#L465
> >>
> >> -Original Message-
> >> From: Arvid Heise 
> >> Sent: Wednesday, May 19, 2021 16:06
> >> To: dev
> >> Subject: Re: [DISCUSS] Watermark propagation with Sink API
> >>
> >>
> >> [EXTERNAL EMAIL]
> >>
> >> Hi Eron,
> >>
> >> Thanks for pushing that topic. I can now see that the benefit is even
> >> bigger than I initially thought. So it's worthwhile anyways to include
> that.
> >>
> >> I also briefly thought about exposing watermarks to all UDFs, but here I
> >> really have an issue to see specific use cases. Could you maybe take a
> few
> >> minutes to think about it as well? I could only see someone misusing
> Async
> >> IO as a sink where a real sink would be more appropriate. In general, if
> >> there is not a clear use case, we shouldn't add the functionality as
> it's
> >> just increased maintenance for no value.
> >>
> >> If we stick to the plan, I think your PR is already in a good shape. We
> >> need to create a FLIP for it though, since it changes Public interfaces
> >> [1]. I was initially not convinced that we should also change the old
> >> SinkFunction interface, but seeing how little the change is, I wouldn't
> >> mind at all to increase consistency. Only when we wrote the FLIP and
> >> approved it (which should be minimal and fast), we should actually look
> at
> >> the PR ;).
> >>
> >> The only thing which I would improve is the name of the function.
> >> processWatermark sounds as if the sink implementer really needs to
> >> implement it (as you would need to do it on a custom operator). I would
> >> make them symmetric to the record writing/invoking method (e.g.
> >> writeWatermark and invokeWatermark).
> >>
> >> As a follow-up PR, we should then migrate KafkaShuffle to the new API.
> >> But that's something I can do.
> >>
> >> [1]
> >>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/Flink*Improvement*Proposals__;Kys!!LpKI!2IQYKfnjRuBgkNRxnPbJeFvTdhWjpwN0urN3m0yz_6W11H74kY5dMnp6nc7o$
> >> [cwiki[.]apache[.]org]
> >>
> >> On Wed, May 19, 2021 at 3:34 AM Eron Wright  >> .invalid>
> >> wrote:
> >>
> >> > Update: opened an issue and a PR.
> >> >
> >> >
> 

Re: [DISCUSS] Watermark propagation with Sink API

[Eron Wright]

Does anyone have further comment on FLIP-167?
https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API

Thanks,
Eron


On Thu, May 20, 2021 at 5:02 PM Eron Wright  wrote:

> Filed FLIP-167: Watermarks for Sink API:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-167%3A+Watermarks+for+Sink+API
>
> I'd like to call a vote next week, is that reasonable?
>
>
> On Wed, May 19, 2021 at 6:28 PM Zhou, Brian  wrote:
>
>> Hi Arvid and Eron,
>>
>> Thanks for the discussion and I read through Eron's pull request and I
>> think this can benefit Pravega Flink connector as well.
>>
>> Here is some background. Pravega had the watermark concept through the
>> event stream since two years ago, and here is a blog introduction[1] for
>> Pravega watermark.
>> Pravega Flink connector also had this watermark integration last year
>> that we wanted to propagate the Flink watermark to Pravega in the
>> SinkFunction, and at that time we just used the existing Flink API that we
>> keep the last watermark in memory and check if watermark changes for each
>> event[2] which is not efficient. With such new interface, we can also
>> manage the watermark propagation much more easily.
>>
>> [1] https://pravega.io/blog/2019/11/08/pravega-watermarking-support/
>> [2]
>> https://github.com/pravega/flink-connectors/blob/master/src/main/java/io/pravega/connectors/flink/FlinkPravegaWriter.java#L465
>>
>> -Original Message-
>> From: Arvid Heise 
>> Sent: Wednesday, May 19, 2021 16:06
>> To: dev
>> Subject: Re: [DISCUSS] Watermark propagation with Sink API
>>
>>
>> [EXTERNAL EMAIL]
>>
>> Hi Eron,
>>
>> Thanks for pushing that topic. I can now see that the benefit is even
>> bigger than I initially thought. So it's worthwhile anyways to include that.
>>
>> I also briefly thought about exposing watermarks to all UDFs, but here I
>> really have an issue to see specific use cases. Could you maybe take a few
>> minutes to think about it as well? I could only see someone misusing Async
>> IO as a sink where a real sink would be more appropriate. In general, if
>> there is not a clear use case, we shouldn't add the functionality as it's
>> just increased maintenance for no value.
>>
>> If we stick to the plan, I think your PR is already in a good shape. We
>> need to create a FLIP for it though, since it changes Public interfaces
>> [1]. I was initially not convinced that we should also change the old
>> SinkFunction interface, but seeing how little the change is, I wouldn't
>> mind at all to increase consistency. Only when we wrote the FLIP and
>> approved it (which should be minimal and fast), we should actually look at
>> the PR ;).
>>
>> The only thing which I would improve is the name of the function.
>> processWatermark sounds as if the sink implementer really needs to
>> implement it (as you would need to do it on a custom operator). I would
>> make them symmetric to the record writing/invoking method (e.g.
>> writeWatermark and invokeWatermark).
>>
>> As a follow-up PR, we should then migrate KafkaShuffle to the new API.
>> But that's something I can do.
>>
>> [1]
>>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/Flink*Improvement*Proposals__;Kys!!LpKI!2IQYKfnjRuBgkNRxnPbJeFvTdhWjpwN0urN3m0yz_6W11H74kY5dMnp6nc7o$
>> [cwiki[.]apache[.]org]
>>
>> On Wed, May 19, 2021 at 3:34 AM Eron Wright > .invalid>
>> wrote:
>>
>> > Update: opened an issue and a PR.
>> >
>> > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLIN
>> > K-22700__;!!LpKI!2IQYKfnjRuBgkNRxnPbJeFvTdhWjpwN0urN3m0yz_6W11H74kY5dM
>> > plbgRO4$ [issues[.]apache[.]org]
>> > https://urldefense.com/v3/__https://github.com/apache/flink/pull/15950
>> > __;!!LpKI!2IQYKfnjRuBgkNRxnPbJeFvTdhWjpwN0urN3m0yz_6W11H74kY5dMtScmG7a
>> > $ [github[.]com]
>> >
>> >
>> > On Tue, May 18, 2021 at 10:03 AM Eron Wright 
>> > wrote:
>> >
>> > > Thanks Arvid and David for sharing your ideas on this subject.  I'm
>> > > glad to hear that you're seeing use cases for watermark propagation
>> > > via an enhanced sink interface.
>> > >
>> > > As you've guessed, my interest is in Pulsar and am exploring some
>> > > options for brokering watermarks across stream processing pipelines.
>> > > I think
>> > Arvid
>> > > is speaking to a high-fidelity solution where the difference between
>> > intra-
>> > > and inter-pipeline flow is eliminated.  My goal is more limited; I
>> > > want
>> > to
>> > > write the watermark that arrives at the sink to Pulsar.  Simply
>> > > imagine that Pulsar has native support for watermarking in its
>> > > producer/consumer API, and we'll leave the details to another forum.
>> > >
>> > > David, I like your invariant.  I see lateness as stemming from the
>> > problem
>> > > domain and from system dynamics (e.g. scheduling, batching, lag).
>> > > When
>> > one
>> > > depends on order-of-observation to generate watermarks, the app may
>> > become
>> > > unduly sensitive to dynamics which bear on 

[VOTE] Release 1.13.1, release candidate #1

[Dawid Wysakowicz]

|Hi everyone,|
|Please review and vote on the release candidate #1 for the version
1.13.1, as follows:|
|[ ] +1, Approve the release|
|[ ] -1, Do not approve the release (please provide specific comments)|
 
 
|The complete staging area is available for your review, which includes:|
|* JIRA release notes [1],|
|* the official Apache source release and binary convenience releases to
be deployed to dist.apache.org [2], which are signed with the key with
fingerprint ||31D2DD10BFC15A2D|| [3],|
|* all artifacts to be deployed to the Maven Central Repository [4],|
|* source code tag "release-1.13.1-rc1" [5],|
|* website pull request listing the new release and adding announcement
blog post [6]. |
 
|The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.|
 
|Best,|
|Dawid|
 
|[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350058|
|[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.1-rc1/|
|[3] https://dist.apache.org/repos/dist/release/flink/KEYS
|
|[4]
https://repository.apache.org/content/repositories/orgapacheflink-1422/|
|[5] https://github.com/apache/flink/tree/release-1.13.1-rc1|
|[6] https://github.com/apache/flink-web/pull/448|


OpenPGP_signature
Description: OpenPGP digital signature

[jira] [Created] (FLINK-22774) Package flink-sql-connector-kinesis with newer Guava version

[Timo Walther (Jira)]

Timo Walther created FLINK-22774:


 Summary: Package flink-sql-connector-kinesis with newer Guava 
version
 Key: FLINK-22774
 URL: https://issues.apache.org/jira/browse/FLINK-22774
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Ecosystem
Reporter: Timo Walther
Assignee: Timo Walther


`flink-coonector-kinesis` uses a very old version (18.0) for compatibility 
reasons. However, since we don't expose Guava and relocate it in SQL 
connectors, we can use a newer version and avoid security concerns raised by 
vulnerability tools.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22773) Optimize the construction of pipelined regions

[Zhilong Hong (Jira)]

Zhilong Hong created FLINK-22773:


 Summary: Optimize the construction of pipelined regions
 Key: FLINK-22773
 URL: https://issues.apache.org/jira/browse/FLINK-22773
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Zhilong Hong


During the initialization of DefaultExecutionTopology, pipelined regions will 
be computed for scheduling. Currently the complexity of this procedure is 
O(N^2):
 
{code:java}
for all vertices in the topology:
  for all consumed results of the vertex:
if the consumed result is reconnectable:
  merge the current region with its producer region
{code}

One possible solution is mentioned in FLINK-17330.

If we can optimize this procedure from O(N^2) to O(N), it will speed up the 
initialization of SchedulerNG, and accelerate the submission of a new job, 
especially for OLAP jobs.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22772) Add TestContext to Python SDK

[Konstantin Knauf (Jira)]

Konstantin Knauf created FLINK-22772:


 Summary: Add TestContext to Python SDK
 Key: FLINK-22772
 URL: https://issues.apache.org/jira/browse/FLINK-22772
 Project: Flink
  Issue Type: Sub-task
  Components: Stateful Functions
Reporter: Konstantin Knauf






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22771) Add TestContext to Java SDK

[Konstantin Knauf (Jira)]

Konstantin Knauf created FLINK-22771:


 Summary: Add TestContext to Java SDK
 Key: FLINK-22771
 URL: https://issues.apache.org/jira/browse/FLINK-22771
 Project: Flink
  Issue Type: Sub-task
  Components: Stateful Functions
Reporter: Konstantin Knauf






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22770) Expose SET/RESET from the parser

[Jira]

Ingo Bürk created FLINK-22770:
-

 Summary: Expose SET/RESET from the parser
 Key: FLINK-22770
 URL: https://issues.apache.org/jira/browse/FLINK-22770
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API, Table SQL / Planner
Affects Versions: 1.13.0
Reporter: Ingo Bürk


The SET/RESET statements (DCL¹) are currently implemented using regular 
expressions in the ExtendedParser. They should instead be properly added to the 
grammar and parser.

In a second step, the ExtendedParser implementation for it can be removed. 
However, this would be a breaking change as key/value would then have to be 
quoted (as they are in other places, i.e. table options) and no longer be 
treated as SQL identifiers.

¹ SET also exists in a no-argument version where it becomes DQL. For the scope 
of this issue I'm excluding this here.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22769) yarnship do not support symbolic directory

[Adrian Zhong (Jira)]

Adrian Zhong created FLINK-22769:


 Summary: yarnship do not support symbolic directory
 Key: FLINK-22769
 URL: https://issues.apache.org/jira/browse/FLINK-22769
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Affects Versions: 1.13.0, 1.12.2, 1.13.1
Reporter: Adrian Zhong


If we pass `-yt ` a symbolic directory, we will get an exception:
{code:java}
Caused by: java.lang.IllegalArgumentException: File to copy cannot be a 
directory: file:/path/to/sql/connectors/Caused by: 
java.lang.IllegalArgumentException: File to copy cannot be a directory: 
file:/path/to/sql/connectors/ at 
org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138) at 
org.apache.flink.yarn.YarnApplicationFileUploader.uploadLocalFileToRemote(YarnApplicationFileUploader.java:197)
 at 
org.apache.flink.yarn.YarnApplicationFileUploader.registerSingleLocalResource(YarnApplicationFileUploader.java:179)
 at 
org.apache.flink.yarn.YarnApplicationFileUploader.registerMultipleLocalResources(YarnApplicationFileUploader.java:289)
 at 
org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:870)
{code}
The root cause is not here but in :
{code:java}
final java.nio.file.Path shipPath = file.toPath();
final java.nio.file.Path parentPath = shipPath.getParent();

Files.walkFileTree(
shipPath.toRealPath(),
new SimpleFileVisitor() {
@Override
public FileVisitResult visitFile(
java.nio.file.Path file, BasicFileAttributes attrs) {
localPaths.add(new Path(file.toUri())); //root cause: localPath 
is an symbolic file, without any file under real path.
relativePaths.add(
new Path(
localResourcesDirectory,
parentPath.relativize(file).toString()));
return FileVisitResult.CONTINUE;
}
});
{code}
Please assign to me, I'd like to be a contributor.

Any other please *do not implement* or create pr before this issue is assigned.

Note: this bug is not os platform specific.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22768) ShuffleMaster supports to promote partitions

[Jin Xing (Jira)]

Jin Xing created FLINK-22768:


 Summary: ShuffleMaster supports to promote partitions
 Key: FLINK-22768
 URL: https://issues.apache.org/jira/browse/FLINK-22768
 Project: Flink
  Issue Type: Sub-task
Reporter: Jin Xing






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Moving to JUnit5

[Till Rohrmann]

Thanks for joining the discussion Qingsheng. In general, I am not opposed
to upgrading our testing library to JUnit 5. Also, the idea of starting
with individual modules and do it incrementally sounds reasonable.

However, before starting to do it like this, the community should agree
that we want to replace JUnit 4 with JUnit 5 eventually. This does not mean
to rewrite all existing tests to use JUnit 5 but at least the default for
all new tests should be JUnit 5 at some point. Otherwise, I fear that we
will fragment the project into modules that use JUnit 5 and those that use
JUnit 4. If this happens then it becomes harder for people to work on the
code base because they always need to know which testing library to use in
which module.

Cheers,
Till

On Tue, May 25, 2021 at 8:53 AM Qingsheng Ren  wrote:

> Hi forks,
>
> I’d like to resume the discussion on migrating to JUnit 5. I’ve been
> working on a connector testing framework and recently have an exploration
> on JUnit 5. I think some features are very helpful for the development of
> the testing framework:
>
> • Extensions
>
> JUnit 5 introduces a new Extension model, which provide a pluggable
> mechanism for extending test classes, like managing test lifecycles and
> providing parameters. Also with the help of extension, we can get rid of
> some limitations introduced by class inheritance, like current TestLogger &
> KafkaTestBase. In testing framework this is helpful for handling the
> lifecycle of Flink cluster and external system.
>
> • Annotations
>
> JUnit 5 provides better support in annotations, working together with
> extensions. We can simple mark types/fields/methods in the test, then let
> extension to search these elements and manage their lifecycle in the test.
> For example test with annotation @MiniCluster will be provided with a
> lifecycle-managed MiniCluster automatically.
>
> • Parameterized tests
>
> JUnit 5 supports more powerful parameterized tests. Testing framework uses
> this to inject different test environments and external contexts into the
> same test case, to run the case under different scenarios.
>
> So I think JUnit 5 is quite flexible for developing such a framework or
> test utility based on it. My suggestion is that we can take connector
> testing framework as a starting point of using JUnit 5, then we can expand
> our exploration to more modules, finally dive into the entire project.
>
> --
> Best Regards,
>
> Qingsheng Ren
> Email: renqs...@gmail.com
> On Dec 1, 2020, 4:54 PM +0800, Khachatryan Roman , wrote:
> > +1 for the migration
> >
> > (I agree with Dawid, for me the most important benefit is better support
> of
> > parameterized tests).
> >
> > Regards,
> > Roman
> >
> >
> > On Mon, Nov 30, 2020 at 9:42 PM Arvid Heise  wrote:
> >
> > > Hi Till,
> > >
> > > immediate benefit would be mostly nested tests for a better test
> structure
> > > and new parameterized tests for less clutter (often test functionality
> is
> > > split into parameterized test and non-parameterized test because of
> JUnit4
> > > limitation). Additionally, having Java8 lambdas to perform fine-grain
> > > exception handling would make all related tests more readable (@Test
> only
> > > allows one exception per test method, while in reality we often have
> more
> > > exceptions / more fine grain assertions and need to resort to
> try-catch --
> > > yuck!). The extension mechanism would also make the mini cluster much
> > > easier to use: we often have to start the cluster manually because of
> > > test-specific configuration, which can be easily avoided in JUnit5.
> > >
> > > In the medium and long-term, I'd also like to use the modular
> > > infrastructure and improved parallelization. The former would allow us
> > > better to implement cross-cutting features like TestLogger (why do we
> need
> > > to extend that manually in every test?). The latter is more relevant
> for
> > > the next push on CI, which would be especially interesting with e2e
> being
> > > available in Java.
> > >
> > > On Mon, Nov 30, 2020 at 2:07 PM Dawid Wysakowicz <
> dwysakow...@apache.org>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Just wanted to express my support for the idea. I did miss certain
> > > > features of JUnit 5 already, an important one being much better
> support
> > > > for parameterized tests.
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 30/11/2020 13:50, Arvid Heise wrote:
> > > > > Hi Chesnay,
> > > > >
> > > > > The vintage runner supports the old annotations, so we don't have
> to
> > > > change
> > > > > them in the first step.
> > > > >
> > > > > The only thing that we need to change are all rules that do not
> extend
> > > > > ExternalResource (e.g., TestWatcher used in TestLogger). This
> change
> > > > needs
> > > > > to be done swiftly as this affects the shared infrastructure as you
> > > have
> > > > > mentioned.
> > > > >
> > > > > Only afterwards, we start to actually migrate the individual tests.
> > > 

[jira] [Created] (FLINK-22767) Optimize the initialization of ExecutionSlotSharingGroupBuilder

[Zhilong Hong (Jira)]

Zhilong Hong created FLINK-22767:


 Summary: Optimize the initialization of 
ExecutionSlotSharingGroupBuilder
 Key: FLINK-22767
 URL: https://issues.apache.org/jira/browse/FLINK-22767
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Zhilong Hong


Based on the scheduler benchmark introduced in FLINK-21731, we find that during 
the initialization of ExecutionSlotSharingGroupBuilder, there's a procedure 
that has O(N^2) complexity: 
{{ExecutionSlotSharingGroupBuilder#tryFindAvailableProducerExecutionSlotSharingGroupFor}}.
 This initialization happens during the initialization of 
LocalInputPreferredSlotSharingStrategy. 

The original implementation is: 
{code:java}
for all SchedulingExecutionVertex in DefaultScheduler:
  for all consumed SchedulingResultPartition of the SchedulingExecutionVertex:
get the result partition's producer vertex and determine the 
ExecutionSlotSharingGroup where the producer vertex locates is available for 
current vertex{code}
This procedure has O(N^2) complexity.

It's obvious that the result partitions in the same ConsumedPartitionGroup have 
the same producer vertex. So we can just iterate over the 
ConsumedPartitionGroups instead of all the consumed partitions. This will 
decrease the complexity from O(N^2) to O(N).

The optimization of this procedure will speed up the initialization of 
DefaultScheduler. It will accelerate the submission of a new job, especially 
for OLAP jobs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22766) Report metrics of KafkaConsumer in Kafka new source

[Qingsheng Ren (Jira)]

Qingsheng Ren created FLINK-22766:
-

 Summary: Report metrics of KafkaConsumer in Kafka new source
 Key: FLINK-22766
 URL: https://issues.apache.org/jira/browse/FLINK-22766
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Qingsheng Ren
 Fix For: 1.14.0, 1.13.1


Currently Kafka new source doesn't register metrics of KafkaConsumer in 
KafkaPartitionSplitReader. These metrics should be added for debugging and 
monitoring purpose. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22765) ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is unstable

[Robert Metzger (Jira)]

Robert Metzger created FLINK-22765:
--

 Summary: ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is 
unstable
 Key: FLINK-22765
 URL: https://issues.apache.org/jira/browse/FLINK-22765
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.14.0
Reporter: Robert Metzger
 Fix For: 1.14.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18292=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56

{code}
May 25 00:56:38 java.lang.AssertionError: 
May 25 00:56:38 
May 25 00:56:38 Expected: is ""
May 25 00:56:38  but: was "The system is out of resources.\nConsult the 
following stack trace for details."
May 25 00:56:38 at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:956)
May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:923)
May 25 00:56:38 at 
org.apache.flink.runtime.util.ExceptionUtilsITCase.run(ExceptionUtilsITCase.java:94)
May 25 00:56:38 at 
org.apache.flink.runtime.util.ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError(ExceptionUtilsITCase.java:70)
May 25 00:56:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
May 25 00:56:38 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
May 25 00:56:38 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
May 25 00:56:38 at java.lang.reflect.Method.invoke(Method.java:498)
May 25 00:56:38 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
May 25 00:56:38 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
May 25 00:56:38 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
May 25 00:56:38 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
May 25 00:56:38 at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
May 25 00:56:38 at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
May 25 00:56:38 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
May 25 00:56:38 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
May 25 00:56:38 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
May 25 00:56:38 at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
May 25 00:56:38 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
May 25 00:56:38 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
May 25 00:56:38 at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
May 25 00:56:38 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
May 25 00:56:38 at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
May 25 00:56:38 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
May 25 00:56:38 at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
May 25 00:56:38 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
May 25 00:56:38 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
May 25 00:56:38 at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
May 25 00:56:38 at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
May 25 00:56:38 at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
May 25 00:56:38 at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
May 25 00:56:38 at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
May 25 00:56:38 

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Moving to JUnit5

[Qingsheng Ren]

Hi forks,

I’d like to resume the discussion on migrating to JUnit 5. I’ve been working on 
a connector testing framework and recently have an exploration on JUnit 5. I 
think some features are very helpful for the development of the testing 
framework:

• Extensions

JUnit 5 introduces a new Extension model, which provide a pluggable mechanism 
for extending test classes, like managing test lifecycles and providing 
parameters. Also with the help of extension, we can get rid of some limitations 
introduced by class inheritance, like current TestLogger & KafkaTestBase. In 
testing framework this is helpful for handling the lifecycle of Flink cluster 
and external system.

• Annotations

JUnit 5 provides better support in annotations, working together with 
extensions. We can simple mark types/fields/methods in the test, then let 
extension to search these elements and manage their lifecycle in the test. For 
example test with annotation @MiniCluster will be provided with a 
lifecycle-managed MiniCluster automatically.

• Parameterized tests

JUnit 5 supports more powerful parameterized tests. Testing framework uses this 
to inject different test environments and external contexts into the same test 
case, to run the case under different scenarios.

So I think JUnit 5 is quite flexible for developing such a framework or test 
utility based on it. My suggestion is that we can take connector testing 
framework as a starting point of using JUnit 5, then we can expand our 
exploration to more modules, finally dive into the entire project.

--
Best Regards,

Qingsheng Ren
Email: renqs...@gmail.com
On Dec 1, 2020, 4:54 PM +0800, Khachatryan Roman , wrote:
> +1 for the migration
>
> (I agree with Dawid, for me the most important benefit is better support of
> parameterized tests).
>
> Regards,
> Roman
>
>
> On Mon, Nov 30, 2020 at 9:42 PM Arvid Heise  wrote:
>
> > Hi Till,
> >
> > immediate benefit would be mostly nested tests for a better test structure
> > and new parameterized tests for less clutter (often test functionality is
> > split into parameterized test and non-parameterized test because of JUnit4
> > limitation). Additionally, having Java8 lambdas to perform fine-grain
> > exception handling would make all related tests more readable (@Test only
> > allows one exception per test method, while in reality we often have more
> > exceptions / more fine grain assertions and need to resort to try-catch --
> > yuck!). The extension mechanism would also make the mini cluster much
> > easier to use: we often have to start the cluster manually because of
> > test-specific configuration, which can be easily avoided in JUnit5.
> >
> > In the medium and long-term, I'd also like to use the modular
> > infrastructure and improved parallelization. The former would allow us
> > better to implement cross-cutting features like TestLogger (why do we need
> > to extend that manually in every test?). The latter is more relevant for
> > the next push on CI, which would be especially interesting with e2e being
> > available in Java.
> >
> > On Mon, Nov 30, 2020 at 2:07 PM Dawid Wysakowicz 
> > wrote:
> >
> > > Hi all,
> > >
> > > Just wanted to express my support for the idea. I did miss certain
> > > features of JUnit 5 already, an important one being much better support
> > > for parameterized tests.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 30/11/2020 13:50, Arvid Heise wrote:
> > > > Hi Chesnay,
> > > >
> > > > The vintage runner supports the old annotations, so we don't have to
> > > change
> > > > them in the first step.
> > > >
> > > > The only thing that we need to change are all rules that do not extend
> > > > ExternalResource (e.g., TestWatcher used in TestLogger). This change
> > > needs
> > > > to be done swiftly as this affects the shared infrastructure as you
> > have
> > > > mentioned.
> > > >
> > > > Only afterwards, we start to actually migrate the individual tests.
> > That
> > > > can be done module by module or as we go. I actually found a nice
> > article
> > > > that leverages the migration assist of IntelliJ [1].
> > > >
> > > > As the last stop, we remove the vintage runner - all JUnit4 tests have
> > > been
> > > > migrated and new tests cannot use old annotation etc. anymore.
> > > >
> > > > [1]
> > > >
> > >
> > https://blog.jetbrains.com/idea/2020/08/migrating-from-junit-4-to-junit-5/
> > > >
> > > > On Mon, Nov 30, 2020 at 1:32 PM Chesnay Schepler 
> > > wrote:
> > > >
> > > > > I presume we cannot do the migration module-wise due to shared test
> > > > > utilities that rely on JUnit interfaces?
> > > > >
> > > > > On 11/30/2020 1:30 PM, Chesnay Schepler wrote:
> > > > > > Is it feasible that 2 people can do the migration within a short
> > > > > > time-frame (say, a week)?
> > > > > > Must the migration of a test be done in one go, or can we for 
> > > > > > example
> > > > > > first rename all the Before/After annotations and then to the rest?
> > > > > > Are there any issues with 

[jira] [Created] (FLINK-22764) Disable Values Source node in streaming plan optimization

[Leonard Xu (Jira)]

Leonard Xu created FLINK-22764:
--

 Summary: Disable Values Source node in streaming plan optimization
 Key: FLINK-22764
 URL: https://issues.apache.org/jira/browse/FLINK-22764
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.13.0
Reporter: Leonard Xu


The `Values` source node may be produced in streaming plan optimization by 
`FlinkPruneEmptyRules` currently,  and the `Values` source will be soon to 
`FINISHED` status which will lead to the whole job can not do checkpoint and 
savepoint.

 

I think we can add an option to enable/disable values source input in streaming 
job to make the job do checkpoint/savepoint normally, for above situation we 
can throw an exception.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)