[jira] [Created] (FLINK-20435) Refactor ExecNode

[godfrey he (Jira)]

godfrey he created FLINK-20435:
--

 Summary: Refactor ExecNode
 Key: FLINK-20435
 URL: https://issues.apache.org/jira/browse/FLINK-20435
 Project: Flink
  Issue Type: Improvement
Reporter: godfrey he


Currently, there are many improvements about ExecNode:
1. simplify type parameter of {{ExecNode}}. Currently, 
{{ExecNode#translateToPlan}} takes {{BatchPlanner}} or {{StreamPlanner}} as a 
parameter, so {{ExecNode}} has a type parameter {{E <: Planner}}, which 
indicates the node is a batch node or a streaming node. While in the future, a 
plan may contain both batch nodes and stream node. The type parameter can be 
removed, the implementation base class can cast the planner to expected planner.
2. port ExecNode to Java
3. separate the implementation of {{RelNode}} and {{ExecNode}}. Currently, an 
execution node extends both from {{RelNode}} and {{ExecNode}}. After a physical 
node is converted to an exec node, many parameters are unnecessary, such as: 
RelOptCluster, RelTraitSet, etc. With more optimizations on {{ExecNode}}, We 
need {{ExecNode}} to be cleaner and simpler. So we will separate the 
implementation of {{RelNode}} and {{ExecNode}}.

This is an umbrella issue, we will create more related sub-tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: flink 1.9 Restore from a checkpoint taken in 1.11

[Yun Tang]

Hi Lu

Flink guarantees backwards compatibility but not forwards compatibility and you 
can find the compatibility table here [1].
Flink-1.11 introduced unaligned checkpoint which upgrades the savepoint version 
to 3, and that's why Flink-1.9 cannot consume savepoint generated by Flink-1.11.


[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/upgrading.html#compatibility-table

Best
Yun Tang


From: Lu Niu 
Sent: Tuesday, December 1, 2020 9:53
To: dev@flink.apache.org 
Subject: flink 1.9 Restore from a checkpoint taken in 1.11

Hi, Flink dev

Is it supported that a flink job in version 1.9 could restore from a
checkpoint taken from the same job using 1.11? The context is we are
migrating to version 1.11 and we need a backup plan for emergency fallback.
We did a test and it throws error:
```
Caused by: org.apache.flink.runtime.rest.util.RestClientException:
[Internal server error., (JobManagerRunner.java:152)
at
org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:83)
at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:387)
at
org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34)
... 7 more
Caused by: java.lang.IllegalArgumentException: Cannot restore savepoint
version 3.
at
org.apache.flink.runtime.checkpoint.savepoint.SavepointSerializers.getSerializer(SavepointSerializers.java:80)
at
org.apache.flink.runtime.checkpoint.Checkpoints.loadCheckpointMetadata(Checkpoints.java:106)
at
org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:143)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1099)
at
org.apache.flink.runtime.scheduler.LegacyScheduler.tryRestoreExecutionGraphFromSavepoint(LegacyScheduler.java:237)
at
org.apache.flink.runtime.scheduler.LegacyScheduler.createAndRestoreExecutionGraph(LegacyScheduler.java:196)
at
org.apache.flink.runtime.scheduler.LegacyScheduler.(LegacyScheduler.java:176)
at
org.apache.flink.runtime.scheduler.LegacySchedulerFactory.createInstance(LegacySchedulerFactory.java:70)
at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278)
at
org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:266)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.(JobManagerRunner.java:146)
... 10 more

End of exception on server side>]
```
So it seems not. I just want to confirm that with the community.


Best
Lu

[jira] [Created] (FLINK-20434) Getting The desired archetype does not exist

[Avi Levi (Jira)]

Avi Levi created FLINK-20434:


 Summary: Getting The desired archetype does not exist
 Key: FLINK-20434
 URL: https://issues.apache.org/jira/browse/FLINK-20434
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Avi Levi


Following the instructions found here 
[https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/datastream_api.html#final-application]
 
{code:java}
// mvn archetype:generate \
-DarchetypeGroupId=org.apache.flink \
-DarchetypeArtifactId=flink-walkthrough-datastream-scala \
-DarchetypeVersion=1.12.0 \
-DgroupId=frauddetection \
-DartifactId=frauddetection \
-Dversion=0.1 \
-Dpackage=spendreport \
-DinteractiveMode=false
{code}
 results :
{code:java}
//[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:3.2.0:generate (default-cli) on 
project standalone-pom: The desired archetype does not exist 
(org.apache.flink:flink-walkthrough-datastream-scala:1.12.0) -> [Help 1]
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Release 1.12.0, release candidate #2

[Robert Metzger]

Hi everyone,

It seems that all blockers for 1.12.0 have been resolved, this is the first
voting release candidate.

Please review and vote on the release candidate #2 for the version 1.12.0,
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1a], and website release notes [1b]
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint D9839159 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.12.0-rc2" [5]

We will soon publish the PR for the release announcement blog post!

The vote will be open for at least 72 hours (Friday 7:35am CET). It is
adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks,
Dian & Robert

[1a]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348263
[1b] https://github.com/apache/flink/pull/14195
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.0-rc2/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1403
[5] https://github.com/apache/flink/releases/tag/release-1.12.0-rc2

[jira] [Created] (FLINK-20433) UnalignedCheckpointTestBase.execute failed with "TestTimedOutException: test timed out after 300 seconds"

[Dian Fu (Jira)]

Dian Fu created FLINK-20433:
---

 Summary: UnalignedCheckpointTestBase.execute failed with 
"TestTimedOutException: test timed out after 300 seconds"
 Key: FLINK-20433
 URL: https://issues.apache.org/jira/browse/FLINK-20433
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.12.0
Reporter: Dian Fu
 Fix For: 1.12.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10353=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56

{code}
2020-12-01T01:33:37.8672135Z [ERROR] execute[Parallel cogroup, p = 
10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase)  Time 
elapsed: 300.308 s  <<< ERROR!
2020-12-01T01:33:37.8672736Z org.junit.runners.model.TestTimedOutException: 
test timed out after 300 seconds
2020-12-01T01:33:37.8673110Zat sun.misc.Unsafe.park(Native Method)
2020-12-01T01:33:37.8673463Zat 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2020-12-01T01:33:37.8673951Zat 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
2020-12-01T01:33:37.8674429Zat 
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
2020-12-01T01:33:37.8686627Zat 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
2020-12-01T01:33:37.8687167Zat 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2020-12-01T01:33:37.8687859Zat 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1842)
2020-12-01T01:33:37.8696554Zat 
org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:70)
2020-12-01T01:33:37.8697226Zat 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1822)
2020-12-01T01:33:37.8697885Zat 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1804)
2020-12-01T01:33:37.8698514Zat 
org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:122)
2020-12-01T01:33:37.8699131Zat 
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.execute(UnalignedCheckpointITCase.java:159)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20432) SQLClientSchemaRegistryITCase hangs

[Dian Fu (Jira)]

Dian Fu created FLINK-20432:
---

 Summary: SQLClientSchemaRegistryITCase hangs
 Key: FLINK-20432
 URL: https://issues.apache.org/jira/browse/FLINK-20432
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.13.0
Reporter: Dian Fu


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10351=logs=739e6eac-8312-5d31-d437-294c4d26fced=a68b8d89-50e9-5977-4500-f4fde4f57f9b

{code}
2020-12-01T01:07:29.6516521Z Dec 01 01:07:29 [INFO] 
---
2020-12-01T01:07:30.5779942Z Dec 01 01:07:30 [INFO] Running 
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase
2020-12-01T01:08:24.8896937Z Dec 01 01:08:24 [INFO] Tests run: 1, Failures: 0, 
Errors: 0, Skipped: 0, Time elapsed: 54.305 s - in 
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase
2020-12-01T01:08:24.8900917Z Dec 01 01:08:24 [INFO] Running 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-12-01T01:09:09.0799444Z Dec 01 01:09:09 [INFO] Tests run: 1, Failures: 0, 
Errors: 0, Skipped: 0, Time elapsed: 44.184 s - in 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-12-01T01:09:09.0825540Z Dec 01 01:09:09 [INFO] Running 
org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase
2020-12-01T01:41:06.5542739Z 
==
2020-12-01T01:41:06.5544689Z === WARNING: This E2E Run will time out in the 
next few minutes. Starting to upload the log output ===
2020-12-01T01:41:06.5549107Z 
==
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20431) KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 expected:<10> but was:<1>

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-20431:


 Summary: 
KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134
 expected:<10> but was:<1>
 Key: FLINK-20431
 URL: https://issues.apache.org/jira/browse/FLINK-20431
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.13.0
Reporter: Huang Xingbo


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10351=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5]
[ERROR] Failures: 
[ERROR] 
KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134
 expected:<10> but was:<1>
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

flink 1.9 Restore from a checkpoint taken in 1.11

[Lu Niu]

Hi, Flink dev

Is it supported that a flink job in version 1.9 could restore from a
checkpoint taken from the same job using 1.11? The context is we are
migrating to version 1.11 and we need a backup plan for emergency fallback.
We did a test and it throws error:
```
Caused by: org.apache.flink.runtime.rest.util.RestClientException:
[Internal server error., (JobManagerRunner.java:152)
at
org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:83)
at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:387)
at
org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34)
... 7 more
Caused by: java.lang.IllegalArgumentException: Cannot restore savepoint
version 3.
at
org.apache.flink.runtime.checkpoint.savepoint.SavepointSerializers.getSerializer(SavepointSerializers.java:80)
at
org.apache.flink.runtime.checkpoint.Checkpoints.loadCheckpointMetadata(Checkpoints.java:106)
at
org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:143)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1099)
at
org.apache.flink.runtime.scheduler.LegacyScheduler.tryRestoreExecutionGraphFromSavepoint(LegacyScheduler.java:237)
at
org.apache.flink.runtime.scheduler.LegacyScheduler.createAndRestoreExecutionGraph(LegacyScheduler.java:196)
at
org.apache.flink.runtime.scheduler.LegacyScheduler.(LegacyScheduler.java:176)
at
org.apache.flink.runtime.scheduler.LegacySchedulerFactory.createInstance(LegacySchedulerFactory.java:70)
at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278)
at
org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:266)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.(JobManagerRunner.java:146)
... 10 more

End of exception on server side>]
```
So it seems not. I just want to confirm that with the community.


Best
Lu

[jira] [Created] (FLINK-20430) Broken links in deployment/index.zh.md

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-20430:


 Summary: Broken links in deployment/index.zh.md
 Key: FLINK-20430
 URL: https://issues.apache.org/jira/browse/FLINK-20430
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.12.0, 1.13.0
Reporter: Huang Xingbo
 Fix For: 1.12.0, 1.13.0


When executing the script `build_docs.sh`, it will throw the following 
exception:
{code:java}
Liquid Exception: Could not find document 
'deployment/resource-providers/standalone/index.md' in tag 'link'. Make sure 
the document exists and the path is correct. in deployment/index.zh.md
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20429) KafkaTableITCase.testKafkaTemporalJoinChangelog failed with unexpected results

[Dian Fu (Jira)]

Dian Fu created FLINK-20429:
---

 Summary: KafkaTableITCase.testKafkaTemporalJoinChangelog failed 
with unexpected results
 Key: FLINK-20429
 URL: https://issues.apache.org/jira/browse/FLINK-20429
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Table SQL / Planner
Affects Versions: 1.13.0
Reporter: Dian Fu
 Fix For: 1.12.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10349=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5

{code}
2020-11-30T18:41:15.9353975Z Test testKafkaTemporalJoinChangelog[legacy = 
false, format = 
csv](org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase) failed 
with:
2020-11-30T18:41:15.9388679Z java.lang.AssertionError: 
expected:<[o_001,2020-10-01T00:01,p_001,1970-01-01T00:00,11.1100,Alice,scooter,1,11.1100,
 
o_002,2020-10-01T00:02,p_002,1970-01-01T00:00,23.1100,Bob,basketball,1,23.1100, 
o_003,2020-10-01T12:00,p_001,2020-10-01T12:00,12.9900,Tom,scooter,2,25.9800, 
o_004,2020-10-01T12:00,p_002,2020-10-01T12:00,19.9900,King,basketball,2,39.9800,
 
o_005,2020-10-01T18:00,p_001,2020-10-01T18:00,11.9900,Leonard,scooter,10,119.9000,
 o_006,2020-10-01T18:00,null,null,null,Leonard,null,10,null]> but 
was:<[o_001,2020-10-01T00:01,p_001,1970-01-01T00:00,11.1100,Alice,scooter,1,11.1100,
 
o_002,2020-10-01T00:02,p_002,1970-01-01T00:00,23.1100,Bob,basketball,1,23.1100, 
o_003,2020-10-01T12:00,p_001,2020-10-01T12:00,12.9900,Tom,scooter,2,25.9800, 
o_004,2020-10-01T12:00,p_002,1970-01-01T00:00,23.1100,King,basketball,2,46.2200,
 
o_005,2020-10-01T18:00,p_001,2020-10-01T18:00,11.9900,Leonard,scooter,10,119.9000,
 o_006,2020-10-01T18:00,null,null,null,Leonard,null,10,null]>
2020-11-30T18:41:15.9392921Zat org.junit.Assert.fail(Assert.java:88)
2020-11-30T18:41:15.9403705Zat 
org.junit.Assert.failNotEquals(Assert.java:834)
2020-11-30T18:41:15.9404670Zat 
org.junit.Assert.assertEquals(Assert.java:118)
2020-11-30T18:41:15.9405477Zat 
org.junit.Assert.assertEquals(Assert.java:144)
2020-11-30T18:41:15.9406471Zat 
org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testKafkaTemporalJoinChangelog(KafkaTableITCase.java:633)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20428) ZooKeeperLeaderElectionConnectionHandlingTest.testConnectionSuspendedHandlingDuringInitialization failed with "No result is expected since there was no leader elected be

[Dian Fu (Jira)]

Dian Fu created FLINK-20428:
---

 Summary: 
ZooKeeperLeaderElectionConnectionHandlingTest.testConnectionSuspendedHandlingDuringInitialization
 failed with "No result is expected since there was no leader elected before 
stopping the server, yet"
 Key: FLINK-20428
 URL: https://issues.apache.org/jira/browse/FLINK-20428
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Dian Fu
 Fix For: 1.12.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10327=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=05b74a19-4ee4-5036-c46f-ada307df6cf0

{code}
2020-11-30T13:22:47.5981970Z [ERROR] 
testConnectionSuspendedHandlingDuringInitialization(org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionConnectionHandlingTest)
  Time elapsed: 2.16 s  <<< FAILURE!
2020-11-30T13:22:47.5984731Z java.lang.AssertionError: 
2020-11-30T13:22:47.5985220Z No result is expected since there was no leader 
elected before stopping the server, yet.
2020-11-30T13:22:47.5985546Z Expected: is null
2020-11-30T13:22:47.5985866Z  but: was 

2020-11-30T13:22:47.5986376Zat 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
2020-11-30T13:22:47.5986752Zat org.junit.Assert.assertThat(Assert.java:956)
2020-11-30T13:22:47.5990853Zat 
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionConnectionHandlingTest.testConnectionSuspendedHandlingDuringInitialization(ZooKeeperLeaderElectionConnectionHandlingTest.java:106)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Moving to JUnit5

[Arvid Heise]

Hi Till,

immediate benefit would be mostly nested tests for a better test structure
and new parameterized tests for less clutter (often test functionality is
split into parameterized test and non-parameterized test because of JUnit4
limitation). Additionally, having Java8 lambdas to perform fine-grain
exception handling would make all related tests more readable (@Test only
allows one exception per test method, while in reality we often have more
exceptions / more fine grain assertions and need to resort to try-catch --
yuck!). The extension mechanism would also make the mini cluster much
easier to use: we often have to start the cluster manually because of
test-specific configuration, which can be easily avoided in JUnit5.

In the medium and long-term, I'd also like to use the modular
infrastructure and improved parallelization. The former would allow us
better to implement cross-cutting features like TestLogger (why do we need
to extend that manually in every test?). The latter is more relevant for
the next push on CI, which would be especially interesting with e2e being
available in Java.

On Mon, Nov 30, 2020 at 2:07 PM Dawid Wysakowicz 
wrote:

> Hi all,
>
> Just wanted to express my support for the idea. I did miss certain
> features of JUnit 5 already, an important one being much better support
> for parameterized tests.
>
> Best,
>
> Dawid
>
> On 30/11/2020 13:50, Arvid Heise wrote:
> > Hi Chesnay,
> >
> > The vintage runner supports the old annotations, so we don't have to
> change
> > them in the first step.
> >
> > The only thing that we need to change are all rules that do not extend
> > ExternalResource (e.g., TestWatcher used in TestLogger). This change
> needs
> > to be done swiftly as this affects the shared infrastructure as you have
> > mentioned.
> >
> > Only afterwards, we start to actually migrate the individual tests. That
> > can be done module by module or as we go. I actually found a nice article
> > that leverages the migration assist of IntelliJ [1].
> >
> > As the last stop, we remove the vintage runner - all JUnit4 tests have
> been
> > migrated and new tests cannot use old annotation etc. anymore.
> >
> > [1]
> >
> https://blog.jetbrains.com/idea/2020/08/migrating-from-junit-4-to-junit-5/
> >
> > On Mon, Nov 30, 2020 at 1:32 PM Chesnay Schepler 
> wrote:
> >
> >> I presume we cannot do the migration module-wise due to shared test
> >> utilities that rely on JUnit interfaces?
> >>
> >> On 11/30/2020 1:30 PM, Chesnay Schepler wrote:
> >>> Is it feasible that 2 people can do the migration within a short
> >>> time-frame (say, a week)?
> >>> Must the migration of a test be done in one go, or can we for example
> >>> first rename all the Before/After annotations and then to the rest?
> >>> Are there any issues with other test dependencies (i.e., hamcrest,
> >>> powermock (PowerMockRunner), mockito) that we should be aware of?
> >>>
> >>> I generally like the idea of using JUnit 5, but am wary of this
> >>> migration dragging on for too long.
> >>>
> >>> On 11/27/2020 3:29 PM, Arvid Heise wrote:
>  Dear devs,
> 
>  I'd like to start a discussion to migrate to a higher JUnit version.
> 
>  The main motivations are:
>  - Making full use of Java 8 Lambdas for writing easier to read tests
>  and a
>  better performing way of composing failure messages.
>  - Improved test structures with nested and dynamic tests.
>  - Much better support for parameterized tests to avoid separating
>  parameterized and non-parameterized parts into different test classes.
>  - Composable dependencies and better hooks for advanced use cases
>  (TestLogger).
>  - Better exception verification
>  - More current infrastructure
>  - Better parallelizable
> 
>  Why now?
>  - JUnit5 is now mature enough to consider it for such a complex
> project
>  - We are porting more and more e2e tests to JUnit and it would be a
>  pity to
>  do all the work twice (okay some already has been done and would
>  result in
>  adjustments, but the sooner we migrate, the less needs to be touched
>  twice)
> 
>  Why JUnit5?
>  There are other interesting alternatives, such as TestNG. I'm happy
>  to hear
>  specific alternatives. For now, I'd like to focus on JUnit4 for an
>  easier
>  migration path.
> 
>  Please discuss if you would also be interested in moving onward. To
> get
>  some overview, I'd like to see some informal +1 for the options:
> 
>  [ ] Stick to JUnit4 for the time being
>  [ ] Move to JUnit5 (see migration path below)
>  [ ] Alternative idea + advantages over JUnit5 + some very rough
>  migration
>  path
> 
>  ---
> 
>  Migrating from JUnit4 to JUnit5 can be done in some steps, so that we
>  can
>  gradually move from JUnit4 to JUnit5.
> 
>  0. (There is a way to use JUnit4 + 5 at the same time in a project -
>  

[jira] [Created] (FLINK-20427) Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to data loss

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-20427:
-

 Summary: Remove CheckpointConfig.setPreferCheckpointForRecovery 
because it can lead to data loss
 Key: FLINK-20427
 URL: https://issues.apache.org/jira/browse/FLINK-20427
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream, Runtime / Checkpointing
Affects Versions: 1.12.0
Reporter: Till Rohrmann
 Fix For: 1.13.0


The {{CheckpointConfig.setPreferCheckpointForRecovery}} allows to configure 
whether Flink prefers checkpoints for recovery if the 
{{CompletedCheckpointStore}} contains savepoints and checkpoints. This is 
problematic because due to this feature, Flink might prefer older checkpoints 
over newer savepoints for recovery. Since some components expect that the 
always the latest checkpoint/savepoint is used (e.g. the 
{{SourceCoordinator}}), it breaks assumptions and can lead to {{SourceSplits}} 
which are not read. This effectively means that the system loses data. 
Similarly, this behaviour can cause that exactly once sinks might output 
results multiple times which violates the processing guarantees. Hence, I 
believe that we should remove this setting because it changes Flink's behaviour 
in some very significant way potentially w/o the user noticing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20426) Broken links to hadoop.md

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-20426:


 Summary: Broken links to hadoop.md
 Key: FLINK-20426
 URL: https://issues.apache.org/jira/browse/FLINK-20426
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz
 Fix For: 1.13.0


In FLINK-20347 we removed the {{hadoop.md}} page, but there are still links in 
other pages:
* dev/project-configuration.md
* dev/batch/hadoop-compatibility.md
* connectors/hive/index.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Moving to JUnit5

[Dawid Wysakowicz]

Hi all,

Just wanted to express my support for the idea. I did miss certain
features of JUnit 5 already, an important one being much better support
for parameterized tests.

Best,

Dawid

On 30/11/2020 13:50, Arvid Heise wrote:
> Hi Chesnay,
>
> The vintage runner supports the old annotations, so we don't have to change
> them in the first step.
>
> The only thing that we need to change are all rules that do not extend
> ExternalResource (e.g., TestWatcher used in TestLogger). This change needs
> to be done swiftly as this affects the shared infrastructure as you have
> mentioned.
>
> Only afterwards, we start to actually migrate the individual tests. That
> can be done module by module or as we go. I actually found a nice article
> that leverages the migration assist of IntelliJ [1].
>
> As the last stop, we remove the vintage runner - all JUnit4 tests have been
> migrated and new tests cannot use old annotation etc. anymore.
>
> [1]
> https://blog.jetbrains.com/idea/2020/08/migrating-from-junit-4-to-junit-5/
>
> On Mon, Nov 30, 2020 at 1:32 PM Chesnay Schepler  wrote:
>
>> I presume we cannot do the migration module-wise due to shared test
>> utilities that rely on JUnit interfaces?
>>
>> On 11/30/2020 1:30 PM, Chesnay Schepler wrote:
>>> Is it feasible that 2 people can do the migration within a short
>>> time-frame (say, a week)?
>>> Must the migration of a test be done in one go, or can we for example
>>> first rename all the Before/After annotations and then to the rest?
>>> Are there any issues with other test dependencies (i.e., hamcrest,
>>> powermock (PowerMockRunner), mockito) that we should be aware of?
>>>
>>> I generally like the idea of using JUnit 5, but am wary of this
>>> migration dragging on for too long.
>>>
>>> On 11/27/2020 3:29 PM, Arvid Heise wrote:
 Dear devs,

 I'd like to start a discussion to migrate to a higher JUnit version.

 The main motivations are:
 - Making full use of Java 8 Lambdas for writing easier to read tests
 and a
 better performing way of composing failure messages.
 - Improved test structures with nested and dynamic tests.
 - Much better support for parameterized tests to avoid separating
 parameterized and non-parameterized parts into different test classes.
 - Composable dependencies and better hooks for advanced use cases
 (TestLogger).
 - Better exception verification
 - More current infrastructure
 - Better parallelizable

 Why now?
 - JUnit5 is now mature enough to consider it for such a complex project
 - We are porting more and more e2e tests to JUnit and it would be a
 pity to
 do all the work twice (okay some already has been done and would
 result in
 adjustments, but the sooner we migrate, the less needs to be touched
 twice)

 Why JUnit5?
 There are other interesting alternatives, such as TestNG. I'm happy
 to hear
 specific alternatives. For now, I'd like to focus on JUnit4 for an
 easier
 migration path.

 Please discuss if you would also be interested in moving onward. To get
 some overview, I'd like to see some informal +1 for the options:

 [ ] Stick to JUnit4 for the time being
 [ ] Move to JUnit5 (see migration path below)
 [ ] Alternative idea + advantages over JUnit5 + some very rough
 migration
 path

 ---

 Migrating from JUnit4 to JUnit5 can be done in some steps, so that we
 can
 gradually move from JUnit4 to JUnit5.

 0. (There is a way to use JUnit4 + 5 at the same time in a project -
 you'd
 use a specific JUnit4 runner to execute JUnit5. I'd like to skip this
 step
 as it would slow down migration significantly)
 1. Use JUnit5 with vintage runner. JUnit4 tests run mostly out of the
 box.
 The most important difference is that only 3 base rules are supported
 and
 the remainder needs to be migrated. Luckily, most of our rules derive
 from
 the supported ExternalResource. So in this step, we would need to
 migrate
 the rules.
 2. Implement new tests in JUnit5.
 3. Soft-migrate old tests in JUnit5. This is mostly a renaming of
 annotation (@Before -> @BeforeEach, etc.). Adjust parameterized tests
 (~400), replace rule usages (~670) with extensions, exception handling
 (~1600 tests), and timeouts (~200). This can be done on a test class by
 test class base and there is no hurry.
 4. Remove vintage runner, once most tests are migrated by doing a final
 push for lesser used modules.

 Let me know what you think and I'm happy to answer all questions.

>>>
>>



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (FLINK-20425) Change exit code in FatalExitExceptionHandler to be different from JvmShutdownSafeguard

[Robert Metzger (Jira)]

Robert Metzger created FLINK-20425:
--

 Summary: Change exit code in FatalExitExceptionHandler to be 
different from JvmShutdownSafeguard
 Key: FLINK-20425
 URL: https://issues.apache.org/jira/browse/FLINK-20425
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Robert Metzger
 Fix For: 1.13.0


Both classes mentioned in the title use the same exit code (-17), making it 
impossible to distinguish the cause of an unexpected Flink JVM exit based on 
the exit code alone.

Maybe we could introduce an Enum that defines all possible exit codes in Flink?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Moving to JUnit5

[Till Rohrmann]

Doing a few required changes swiftly and then porting the remaining tests
bit by bit sounds like an approachable plan. However, it will still cost us
some effort I guess. What exactly would be the benefit of this change other
than streamlining a few things? Are there things which we cannot do at the
moment but which would be possible with Junit5?

Don't get me wrong, personally I also would like to use the latest Junit
version with the cool new features available. However, at the end of the
day, we only have limited time and should choose wisely where we invest it.
For example, currently I consider the state of the e2e tests much more
severe and would like to clean this one up first. If you believe that the
community can do both things, then I am fine with it. Maybe the answer
could be that we will never port all existing tests but we offer the
opportunity to write new tests against Junit5. Or maybe porting the
existing tests is so simple that it can be done by a couple of persons in
one go.

Cheers,
Till

On Mon, Nov 30, 2020 at 1:51 PM Arvid Heise  wrote:

> Hi Chesnay,
>
> The vintage runner supports the old annotations, so we don't have to change
> them in the first step.
>
> The only thing that we need to change are all rules that do not extend
> ExternalResource (e.g., TestWatcher used in TestLogger). This change needs
> to be done swiftly as this affects the shared infrastructure as you have
> mentioned.
>
> Only afterwards, we start to actually migrate the individual tests. That
> can be done module by module or as we go. I actually found a nice article
> that leverages the migration assist of IntelliJ [1].
>
> As the last stop, we remove the vintage runner - all JUnit4 tests have been
> migrated and new tests cannot use old annotation etc. anymore.
>
> [1]
> https://blog.jetbrains.com/idea/2020/08/migrating-from-junit-4-to-junit-5/
>
> On Mon, Nov 30, 2020 at 1:32 PM Chesnay Schepler 
> wrote:
>
> > I presume we cannot do the migration module-wise due to shared test
> > utilities that rely on JUnit interfaces?
> >
> > On 11/30/2020 1:30 PM, Chesnay Schepler wrote:
> > > Is it feasible that 2 people can do the migration within a short
> > > time-frame (say, a week)?
> > > Must the migration of a test be done in one go, or can we for example
> > > first rename all the Before/After annotations and then to the rest?
> > > Are there any issues with other test dependencies (i.e., hamcrest,
> > > powermock (PowerMockRunner), mockito) that we should be aware of?
> > >
> > > I generally like the idea of using JUnit 5, but am wary of this
> > > migration dragging on for too long.
> > >
> > > On 11/27/2020 3:29 PM, Arvid Heise wrote:
> > >> Dear devs,
> > >>
> > >> I'd like to start a discussion to migrate to a higher JUnit version.
> > >>
> > >> The main motivations are:
> > >> - Making full use of Java 8 Lambdas for writing easier to read tests
> > >> and a
> > >> better performing way of composing failure messages.
> > >> - Improved test structures with nested and dynamic tests.
> > >> - Much better support for parameterized tests to avoid separating
> > >> parameterized and non-parameterized parts into different test classes.
> > >> - Composable dependencies and better hooks for advanced use cases
> > >> (TestLogger).
> > >> - Better exception verification
> > >> - More current infrastructure
> > >> - Better parallelizable
> > >>
> > >> Why now?
> > >> - JUnit5 is now mature enough to consider it for such a complex
> project
> > >> - We are porting more and more e2e tests to JUnit and it would be a
> > >> pity to
> > >> do all the work twice (okay some already has been done and would
> > >> result in
> > >> adjustments, but the sooner we migrate, the less needs to be touched
> > >> twice)
> > >>
> > >> Why JUnit5?
> > >> There are other interesting alternatives, such as TestNG. I'm happy
> > >> to hear
> > >> specific alternatives. For now, I'd like to focus on JUnit4 for an
> > >> easier
> > >> migration path.
> > >>
> > >> Please discuss if you would also be interested in moving onward. To
> get
> > >> some overview, I'd like to see some informal +1 for the options:
> > >>
> > >> [ ] Stick to JUnit4 for the time being
> > >> [ ] Move to JUnit5 (see migration path below)
> > >> [ ] Alternative idea + advantages over JUnit5 + some very rough
> > >> migration
> > >> path
> > >>
> > >> ---
> > >>
> > >> Migrating from JUnit4 to JUnit5 can be done in some steps, so that we
> > >> can
> > >> gradually move from JUnit4 to JUnit5.
> > >>
> > >> 0. (There is a way to use JUnit4 + 5 at the same time in a project -
> > >> you'd
> > >> use a specific JUnit4 runner to execute JUnit5. I'd like to skip this
> > >> step
> > >> as it would slow down migration significantly)
> > >> 1. Use JUnit5 with vintage runner. JUnit4 tests run mostly out of the
> > >> box.
> > >> The most important difference is that only 3 base rules are supported
> > >> and
> > >> the remainder needs to be migrated. Luckily, most of our rules 

Re: [DISCUSS] Moving to JUnit5

[Arvid Heise]

Hi Chesnay,

The vintage runner supports the old annotations, so we don't have to change
them in the first step.

The only thing that we need to change are all rules that do not extend
ExternalResource (e.g., TestWatcher used in TestLogger). This change needs
to be done swiftly as this affects the shared infrastructure as you have
mentioned.

Only afterwards, we start to actually migrate the individual tests. That
can be done module by module or as we go. I actually found a nice article
that leverages the migration assist of IntelliJ [1].

As the last stop, we remove the vintage runner - all JUnit4 tests have been
migrated and new tests cannot use old annotation etc. anymore.

[1]
https://blog.jetbrains.com/idea/2020/08/migrating-from-junit-4-to-junit-5/

On Mon, Nov 30, 2020 at 1:32 PM Chesnay Schepler  wrote:

> I presume we cannot do the migration module-wise due to shared test
> utilities that rely on JUnit interfaces?
>
> On 11/30/2020 1:30 PM, Chesnay Schepler wrote:
> > Is it feasible that 2 people can do the migration within a short
> > time-frame (say, a week)?
> > Must the migration of a test be done in one go, or can we for example
> > first rename all the Before/After annotations and then to the rest?
> > Are there any issues with other test dependencies (i.e., hamcrest,
> > powermock (PowerMockRunner), mockito) that we should be aware of?
> >
> > I generally like the idea of using JUnit 5, but am wary of this
> > migration dragging on for too long.
> >
> > On 11/27/2020 3:29 PM, Arvid Heise wrote:
> >> Dear devs,
> >>
> >> I'd like to start a discussion to migrate to a higher JUnit version.
> >>
> >> The main motivations are:
> >> - Making full use of Java 8 Lambdas for writing easier to read tests
> >> and a
> >> better performing way of composing failure messages.
> >> - Improved test structures with nested and dynamic tests.
> >> - Much better support for parameterized tests to avoid separating
> >> parameterized and non-parameterized parts into different test classes.
> >> - Composable dependencies and better hooks for advanced use cases
> >> (TestLogger).
> >> - Better exception verification
> >> - More current infrastructure
> >> - Better parallelizable
> >>
> >> Why now?
> >> - JUnit5 is now mature enough to consider it for such a complex project
> >> - We are porting more and more e2e tests to JUnit and it would be a
> >> pity to
> >> do all the work twice (okay some already has been done and would
> >> result in
> >> adjustments, but the sooner we migrate, the less needs to be touched
> >> twice)
> >>
> >> Why JUnit5?
> >> There are other interesting alternatives, such as TestNG. I'm happy
> >> to hear
> >> specific alternatives. For now, I'd like to focus on JUnit4 for an
> >> easier
> >> migration path.
> >>
> >> Please discuss if you would also be interested in moving onward. To get
> >> some overview, I'd like to see some informal +1 for the options:
> >>
> >> [ ] Stick to JUnit4 for the time being
> >> [ ] Move to JUnit5 (see migration path below)
> >> [ ] Alternative idea + advantages over JUnit5 + some very rough
> >> migration
> >> path
> >>
> >> ---
> >>
> >> Migrating from JUnit4 to JUnit5 can be done in some steps, so that we
> >> can
> >> gradually move from JUnit4 to JUnit5.
> >>
> >> 0. (There is a way to use JUnit4 + 5 at the same time in a project -
> >> you'd
> >> use a specific JUnit4 runner to execute JUnit5. I'd like to skip this
> >> step
> >> as it would slow down migration significantly)
> >> 1. Use JUnit5 with vintage runner. JUnit4 tests run mostly out of the
> >> box.
> >> The most important difference is that only 3 base rules are supported
> >> and
> >> the remainder needs to be migrated. Luckily, most of our rules derive
> >> from
> >> the supported ExternalResource. So in this step, we would need to
> >> migrate
> >> the rules.
> >> 2. Implement new tests in JUnit5.
> >> 3. Soft-migrate old tests in JUnit5. This is mostly a renaming of
> >> annotation (@Before -> @BeforeEach, etc.). Adjust parameterized tests
> >> (~400), replace rule usages (~670) with extensions, exception handling
> >> (~1600 tests), and timeouts (~200). This can be done on a test class by
> >> test class base and there is no hurry.
> >> 4. Remove vintage runner, once most tests are migrated by doing a final
> >> push for lesser used modules.
> >>
> >> Let me know what you think and I'm happy to answer all questions.
> >>
> >
> >
>
>

-- 

Arvid Heise | Senior Java Developer



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng

Re: [DISCUSS] Moving to JUnit5

[Chesnay Schepler]

I presume we cannot do the migration module-wise due to shared test 
utilities that rely on JUnit interfaces?


On 11/30/2020 1:30 PM, Chesnay Schepler wrote:
Is it feasible that 2 people can do the migration within a short 
time-frame (say, a week)?
Must the migration of a test be done in one go, or can we for example 
first rename all the Before/After annotations and then to the rest?
Are there any issues with other test dependencies (i.e., hamcrest, 
powermock (PowerMockRunner), mockito) that we should be aware of?


I generally like the idea of using JUnit 5, but am wary of this 
migration dragging on for too long.


On 11/27/2020 3:29 PM, Arvid Heise wrote:

Dear devs,

I'd like to start a discussion to migrate to a higher JUnit version.

The main motivations are:
- Making full use of Java 8 Lambdas for writing easier to read tests 
and a

better performing way of composing failure messages.
- Improved test structures with nested and dynamic tests.
- Much better support for parameterized tests to avoid separating
parameterized and non-parameterized parts into different test classes.
- Composable dependencies and better hooks for advanced use cases
(TestLogger).
- Better exception verification
- More current infrastructure
- Better parallelizable

Why now?
- JUnit5 is now mature enough to consider it for such a complex project
- We are porting more and more e2e tests to JUnit and it would be a 
pity to
do all the work twice (okay some already has been done and would 
result in
adjustments, but the sooner we migrate, the less needs to be touched 
twice)


Why JUnit5?
There are other interesting alternatives, such as TestNG. I'm happy 
to hear
specific alternatives. For now, I'd like to focus on JUnit4 for an 
easier

migration path.

Please discuss if you would also be interested in moving onward. To get
some overview, I'd like to see some informal +1 for the options:

[ ] Stick to JUnit4 for the time being
[ ] Move to JUnit5 (see migration path below)
[ ] Alternative idea + advantages over JUnit5 + some very rough 
migration

path

---

Migrating from JUnit4 to JUnit5 can be done in some steps, so that we 
can

gradually move from JUnit4 to JUnit5.

0. (There is a way to use JUnit4 + 5 at the same time in a project - 
you'd
use a specific JUnit4 runner to execute JUnit5. I'd like to skip this 
step

as it would slow down migration significantly)
1. Use JUnit5 with vintage runner. JUnit4 tests run mostly out of the 
box.
The most important difference is that only 3 base rules are supported 
and
the remainder needs to be migrated. Luckily, most of our rules derive 
from
the supported ExternalResource. So in this step, we would need to 
migrate

the rules.
2. Implement new tests in JUnit5.
3. Soft-migrate old tests in JUnit5. This is mostly a renaming of
annotation (@Before -> @BeforeEach, etc.). Adjust parameterized tests
(~400), replace rule usages (~670) with extensions, exception handling
(~1600 tests), and timeouts (~200). This can be done on a test class by
test class base and there is no hurry.
4. Remove vintage runner, once most tests are migrated by doing a final
push for lesser used modules.

Let me know what you think and I'm happy to answer all questions.

Re: [DISCUSS] Moving to JUnit5

[Chesnay Schepler]

Is it feasible that 2 people can do the migration within a short 
time-frame (say, a week)?
Must the migration of a test be done in one go, or can we for example 
first rename all the Before/After annotations and then to the rest?
Are there any issues with other test dependencies (i.e., hamcrest, 
powermock (PowerMockRunner), mockito) that we should be aware of?


I generally like the idea of using JUnit 5, but am wary of this 
migration dragging on for too long.


On 11/27/2020 3:29 PM, Arvid Heise wrote:

Dear devs,

I'd like to start a discussion to migrate to a higher JUnit version.

The main motivations are:
- Making full use of Java 8 Lambdas for writing easier to read tests and a
better performing way of composing failure messages.
- Improved test structures with nested and dynamic tests.
- Much better support for parameterized tests to avoid separating
parameterized and non-parameterized parts into different test classes.
- Composable dependencies and better hooks for advanced use cases
(TestLogger).
- Better exception verification
- More current infrastructure
- Better parallelizable

Why now?
- JUnit5 is now mature enough to consider it for such a complex project
- We are porting more and more e2e tests to JUnit and it would be a pity to
do all the work twice (okay some already has been done and would result in
adjustments, but the sooner we migrate, the less needs to be touched twice)

Why JUnit5?
There are other interesting alternatives, such as TestNG. I'm happy to hear
specific alternatives. For now, I'd like to focus on JUnit4 for an easier
migration path.

Please discuss if you would also be interested in moving onward. To get
some overview, I'd like to see some informal +1 for the options:

[ ] Stick to JUnit4 for the time being
[ ] Move to JUnit5 (see migration path below)
[ ] Alternative idea + advantages over JUnit5 + some very rough migration
path

---

Migrating from JUnit4 to JUnit5 can be done in some steps, so that we can
gradually move from JUnit4 to JUnit5.

0. (There is a way to use JUnit4 + 5 at the same time in a project - you'd
use a specific JUnit4 runner to execute JUnit5. I'd like to skip this step
as it would slow down migration significantly)
1. Use JUnit5 with vintage runner. JUnit4 tests run mostly out of the box.
The most important difference is that only 3 base rules are supported and
the remainder needs to be migrated. Luckily, most of our rules derive from
the supported ExternalResource. So in this step, we would need to migrate
the rules.
2. Implement new tests in JUnit5.
3. Soft-migrate old tests in JUnit5. This is mostly a renaming of
annotation (@Before -> @BeforeEach, etc.). Adjust parameterized tests
(~400), replace rule usages (~670) with extensions, exception handling
(~1600 tests), and timeouts (~200). This can be done on a test class by
test class base and there is no hurry.
4. Remove vintage runner, once most tests are migrated by doing a final
push for lesser used modules.

Let me know what you think and I'm happy to answer all questions.

[jira] [Created] (FLINK-20424) The percent of acknowledged checkpoint seems incorrect

[zlzhang0122 (Jira)]

zlzhang0122 created FLINK-20424:
---

 Summary: The percent of acknowledged checkpoint seems incorrect
 Key: FLINK-20424
 URL: https://issues.apache.org/jira/browse/FLINK-20424
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Reporter: zlzhang0122
 Attachments: 2020-11-30 14-18-34 的屏幕截图.png

As the picture below, the percent of acknowledged checkpoint seems incorrect.I 
think the number must not be 100% because one of the checkpoint acknowledge was 
failed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20423) Remove usage of {{site.baseurl}} from markdown files

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-20423:


 Summary: Remove usage of {{site.baseurl}} from markdown files
 Key: FLINK-20423
 URL: https://issues.apache.org/jira/browse/FLINK-20423
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Dawid Wysakowicz






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20422) Remove {{ site.baseurl }} from .html files in flink documentation

[Xiao Huang (Jira)]

Xiao Huang created FLINK-20422:
--

 Summary: Remove {{ site.baseurl }} from .html files in flink 
documentation
 Key: FLINK-20422
 URL: https://issues.apache.org/jira/browse/FLINK-20422
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Xiao Huang


Remove the \{\{site.baseurl\}\} from .html files. E.g. 
{{_layouts/redirect.html}}.
This ticket is to solve the remaining problem in 
https://github.com/apache/flink/pull/14235 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20421) Support canal-protobuf format

[Jark Wu (Jira)]

Jark Wu created FLINK-20421:
---

 Summary: Support canal-protobuf format
 Key: FLINK-20421
 URL: https://issues.apache.org/jira/browse/FLINK-20421
 Project: Flink
  Issue Type: Sub-task
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table 
SQL / Ecosystem
Reporter: Jark Wu


Canal also supports encoding data in protobuf format. Protobuf is a more 
efficient format than JSON. It would be great if we can support to interpret 
Canal data in protobuf format. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20420) ES6 ElasticsearchSinkITCase failed due to no output for 900 seconds

[Yun Tang (Jira)]

Yun Tang created FLINK-20420:


 Summary: ES6 ElasticsearchSinkITCase failed due to no output for 
900 seconds
 Key: FLINK-20420
 URL: https://issues.apache.org/jira/browse/FLINK-20420
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch
Affects Versions: 1.12.0
Reporter: Yun Tang


Instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10249=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=18821


{code:java}
Process produced no output for 900 seconds.
==
==
The following Java processes are running (JPS)
==
Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
2274 Launcher
18260 Jps
15916 surefirebooter3434370240444055571.jar
==


"main" #1 prio=5 os_prio=0 tid=0x7feec000b800 nid=0x3e2d runnable 
[0x7feec8541000]
   java.lang.Thread.State: RUNNABLE
at org.testcontainers.shaded.okio.Buffer.indexOf(Buffer.java:1463)
at 
org.testcontainers.shaded.okio.RealBufferedSource.indexOf(RealBufferedSource.java:352)
at 
org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
at 
org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:224)
at 
org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:489)
at 
org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
at 
org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
at 
org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
at 
org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
at 
org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
at 
org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
at 
org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
at 
org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
at 
org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
at org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
at 
org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:280)
at 
org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
at 
org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$87/1112018050.close(Unknown
 Source)
at 
com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
at 
org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:177)
at 
org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:203)
- locked <0x88fcbbf0> (a [Ljava.lang.Object;)
at 
org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
at 
org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
at 
org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
- locked <0x88fcb940> (a 
org.testcontainers.images.LocalImagesCache)
at 
org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
at 
org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
at 
org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:66)
at 
org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:27)
at 
org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
- locked <0x890763d0> (a 
java.util.concurrent.atomic.AtomicReference)
at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
at 
org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1276)
at 
org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
at 
org.testcontainers.elasticsearch.ElasticsearchContainer.(ElasticsearchContainer.java:73)
at 
org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSinkITCase.(ElasticsearchSinkITCase.java:50)
at 

[jira] [Created] (FLINK-20419) Insert fails due to failure to generate execution plan

[Rui Li (Jira)]

Rui Li created FLINK-20419:
--

 Summary: Insert fails due to failure to generate execution plan
 Key: FLINK-20419
 URL: https://issues.apache.org/jira/browse/FLINK-20419
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Rui Li


Test case to reproduce:
{code}
@Test
public void test() throws Exception {
tableEnv.executeSql("create table src(x int)");
tableEnv.executeSql("create table dest(x int) partitioned by (p 
string,q string)");
tableEnv.executeSql("insert into dest select x,'0','0' from src 
order by x").await();
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20418) NPE in IteratorSourceReader

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-20418:
-

 Summary: NPE in IteratorSourceReader
 Key: FLINK-20418
 URL: https://issues.apache.org/jira/browse/FLINK-20418
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.12.0
Reporter: Roman Khachatryan


{code}
@Test
public void testNpe() throws Exception {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
env.setRestartStrategy(new NoRestartStrategyConfiguration());
env.enableCheckpointing(50, CheckpointingMode.EXACTLY_ONCE);
env
.fromSequence(0, 100)
.map(x -> {
Thread.sleep(10);
return x;
})
.addSink(new DiscardingSink<>());
env.execute();
}
{code}

{code}
...
Caused by: java.lang.Exception: Could not perform checkpoint 1 for operator 
Source: Sequence Source -> Map -> Sink: Unnamed (1/1)#0.
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:866)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$triggerCheckpointAsync$8(StreamTask.java:831)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:575)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:539)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not 
complete snapshot 1 for operator Source: Sequence Source -> Map -> Sink: 
Unnamed (1/1)#0. Failure reason: Checkpoint was declined.
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:226)
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:158)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:343)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointStreamOperator(SubtaskCheckpointCoordinatorImpl.java:603)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.buildOperatorSnapshotFutures(SubtaskCheckpointCoordinatorImpl.java:529)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:496)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:266)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$9(StreamTask.java:924)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:914)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:857)
... 10 more
Caused by: java.lang.NullPointerException
at 
org.apache.flink.api.connector.source.lib.util.IteratorSourceReader.snapshotState(IteratorSourceReader.java:132)
at 
org.apache.flink.streaming.api.operators.SourceOperator.snapshotState(SourceOperator.java:264)
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:197)
... 20 more


{code}

cc: [~sewen]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20417) Handle "Too old resource version" exception in Kubernetes watch more gracefully

[Yang Wang (Jira)]

Yang Wang created FLINK-20417:
-

 Summary: Handle "Too old resource version" exception in Kubernetes 
watch more gracefully
 Key: FLINK-20417
 URL: https://issues.apache.org/jira/browse/FLINK-20417
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Reporter: Yang Wang


Currently, when the watcher(pods watcher, configmap watcher) is closed with 
exception, we will call {{WatchCallbackHandler#handleFatalError}}. And this 
could cause JobManager terminating and then failover.

For most cases, this is correct. But not for "too old resource version" 
exception. See more information here[1]. Usually this exception could happen 
when the APIServer is restarted. And we just need to create a new watch and 
continue to do the pods/configmap watching. This could help the Flink cluster 
reducing the impact of K8s cluster restarting.

 

The issue is inspired by this technical article[2]. Thanks the guys from 
tencent for the debugging. Note this is a Chinese documentation.

 

[1]. 
[https://stackoverflow.com/questions/61409596/kubernetes-too-old-resource-version]

[2]. [https://cloud.tencent.com/developer/article/1731416]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20416) Need a cached catalog for batch SQL job

[Sebastian Liu (Jira)]

Sebastian Liu created FLINK-20416:
-

 Summary: Need a cached catalog for batch SQL job
 Key: FLINK-20416
 URL: https://issues.apache.org/jira/browse/FLINK-20416
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API, Table SQL / Planner
Reporter: Sebastian Liu


For OLAP scenarios, There are usually some analytical queries which running 
time is relatively short. These queries are also sensitive to latency. In the 
current Blink sql processing, parse/validate/optimize stages are all need meta 
data from catalog API. But each request to the catalog requires re-run of the 
underlying meta query. 

 

We may need a cached catalog which can cache the table schema and statistic 
info to avoid unnecessary repeated meta requests. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)