Re: [VOTE][FLIP-195] Improve the name and structure of vertex and operator name for job

[Yangze Guo]

+1 (binding)

Also, this will further slim down the TaskDeploymentDescriptor.

Best,
Yangze Guo

On Wed, Nov 24, 2021 at 3:57 PM Yun Tang  wrote:
>
> +1 (binding)
>
> Best
> Yun Tang
>
> On 2021/11/24 07:50:15 Sergey Nuyanzin wrote:
> > +1 (non-binding)
> >
> > On Wed, Nov 24, 2021 at 8:38 AM godfrey he  wrote:
> >
> > > +1 (binding)
> > >
> > > Best,
> > > Godfrey
> > >
> > > Jark Wu  于2021年11月24日周三 下午12:02写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > Btw, @JingZhang I think your vote can be counted into binding now.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Tue, 23 Nov 2021 at 20:19, Jing Zhang  wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Jing Zhang
> > > > >
> > > > > Martijn Visser  于2021年11月23日周二 下午7:42写道：
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Tue, 23 Nov 2021 at 12:13, Aitozi  wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Best,
> > > > > > > Aitozi
> > > > > > >
> > > > > > > wenlong.lwl  于2021年11月23日周二 下午4:00写道：
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > Based on the discussion[1], we seem to have consensus, so I 
> > > > > > > > would
> > > > > like
> > > > > > to
> > > > > > > > start a vote on FLIP-195 [2].
> > > > > > > > Thanks for all of your feedback.
> > > > > > > >
> > > > > > > > The vote will last for at least 72 hours (Nov 26th 16:00 GMT)
> > > unless
> > > > > > > > there is an objection or insufficient votes.
> > > > > > > >
> > > > > > > > [1]
> > > https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
> > > > > > > > [2]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Wenlong Lyu
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
> >

[jira] [Created] (FLINK-25030) Unexpected record in KafkaSourceITCase$IntegrationTests.testMultipleSplits

[Matthias (Jira)]

Matthias created FLINK-25030:


 Summary: Unexpected record in 
KafkaSourceITCase$IntegrationTests.testMultipleSplits
 Key: FLINK-25030
 URL: https://issues.apache.org/jira/browse/FLINK-25030
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.1
Reporter: Matthias


We experienced a test failure in 
{{KafkaSourceITCase$IntegrationTests.testMultipleSplits}} in our Flink fork for 
1.14 due to an unexpected record:
{code}
[...]
Nov 23 21:10:19 [ERROR] 
org.apache.flink.connector.kafka.source.KafkaSourceITCase$IntegrationTests.testMultipleSplits{TestEnvironment,
 ExternalContext}[1]
Nov 23 21:10:19 [ERROR]   Run 1: 
KafkaSourceITCase$IntegrationTests>SourceTestSuiteBase.testMultipleSplits:160 
Nov 23 21:10:19 Expected: Records consumed by Flink should be identical to test 
data and preserve the order in multiple splits
Nov 23 21:10:19  but: Unexpected record '2-13N3fae7bfL1iEMF3I0TaWGC57vrflv' 
at position 367
Nov 23 21:10:19 Current progress of multiple split test data validation:
Nov 23 21:10:19 Split 0 (115/115): 
Nov 23 21:10:19 0-C7bHGoulUrqjQqGM8PiVI6BS9B3Okq2PJdf3EBas3G
Nov 23 21:10:19 0-GRt5T5YYDsgq1t0UBt3cUjvnktIbz
[...]
{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25031) Job finishes iff all job vertices finish

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25031:
--

 Summary: Job finishes iff all job vertices finish
 Key: FLINK-25031
 URL: https://issues.apache.org/jira/browse/FLINK-25031
 Project: Flink
  Issue Type: Sub-task
Reporter: Lijie Wang


The adaptive batch job scheduler needs to build ExecutionGraph dynamically. For 
a dynamic graph, since its execution vertices can be lazily created, a job 
should not finish when all ExecutionVertex(es) finish. Changes should be made 
to let a job finish only when all registered ExecutionJobVertex have finished.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[VOTE][FLIP-195] Improve the name and structure of vertex and operator name for job

[Xianxun Ye]

+1 (non-binding)




On 11/24/2021 15:50，Sergey Nuyanzin wrote：
+1 (non-binding)

On Wed, Nov 24, 2021 at 8:38 AM godfrey he  wrote:

+1 (binding)

Best,
Godfrey

Jark Wu  于2021年11月24日周三 下午12:02写道：

+1 (binding)

Btw, @JingZhang I think your vote can be counted into binding now.

Best,
Jark

On Tue, 23 Nov 2021 at 20:19, Jing Zhang  wrote:

+1 (non-binding)

Best,
Jing Zhang

Martijn Visser  于2021年11月23日周二 下午7:42写道：

+1 (non-binding)

On Tue, 23 Nov 2021 at 12:13, Aitozi  wrote:

+1 (non-binding)

Best,
Aitozi

wenlong.lwl  于2021年11月23日周二 下午4:00写道：

Hi everyone,

Based on the discussion[1], we seem to have consensus, so I would
like
to
start a vote on FLIP-195 [2].
Thanks for all of your feedback.

The vote will last for at least 72 hours (Nov 26th 16:00 GMT)
unless
there is an objection or insufficient votes.

[1]
https://lists.apache.org/thread/kvdxr8db0l5s6wk7hwlt0go5fms99b8t
[2]





https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job

Best,
Wenlong Lyu







--
Best regards,
Sergey

[jira] [Created] (FLINK-25032) Allow to create execution vertices and execution edges lazily

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25032:
--

 Summary: Allow to create execution vertices and execution edges 
lazily
 Key: FLINK-25032
 URL: https://issues.apache.org/jira/browse/FLINK-25032
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Lijie Wang


For a dynamic graph, its execution vertices and execution edges should be 
lazily created.

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25033) Let some scheduler components updatable.

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25033:
--

 Summary: Let some scheduler components updatable.
 Key: FLINK-25033
 URL: https://issues.apache.org/jira/browse/FLINK-25033
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Lijie Wang


Many scheduler components rely on the execution topology to make decisions. 
Some of them will build up some mappings against the execution topology on 
initialization for later use. When the execution topology becomes dynamic, 
these components need to be notified about the topology changes and adjust 
themselves accordingly. These components are:
 * DefaultExecutionTopology
 * SchedulingStrategy
 * PartitionReleaseStrategy
 * SlotSharingStrategy
 * OperatorCoordinatorHandler
 * Network memory of SlotSharingGroup.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25034) Support flexible number of subpartitions in IntermediateResultPartition

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25034:
--

 Summary: Support flexible number of subpartitions in 
IntermediateResultPartition
 Key: FLINK-25034
 URL: https://issues.apache.org/jira/browse/FLINK-25034
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Lijie Wang


Currently, when a task is deployed, it needs to know the parallelism of its 
consumer job vertex. This is because the consumer vertex parallelism is needed 
to decide the _numberOfSubpartitions_ of _PartitionDescriptor_ which is part of 
the {_}ResultPartitionDeploymentDescriptor{_}. The reason behind that is, at 
the moment, for one result partition, different subpartitions serve different 
consumer execution vertices. More specifically, one consumer execution vertex 
only consumes data from subpartition with the same index. 

Considering a dynamic graph, the parallelism of a job vertex may not have been 
decided when its upstream vertices are deployed. To enable Flink to work in 
this case, we need a way to allow an execution vertex to run without knowing 
the parallelism of its consumer job vertices. One basic idea is to enable 
multiple subpartitions in one result partition to serve the same consumer 
execution vertex.

To achieve this goal, we can set the number of subpartitions to be the *max 
parallelism* of the consumer job vertex. When the consumer vertex is deployed, 
it should be assigned with a subpartition range to consume.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25035) Shuffle Service Supports Consuming Subpartition Range

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25035:
--

 Summary: Shuffle Service Supports Consuming Subpartition Range
 Key: FLINK-25035
 URL: https://issues.apache.org/jira/browse/FLINK-25035
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Lijie Wang


In adaptive batch job scheduler, the shuffle service needs to support a 
SingleInputGate to consume  a certain range of subpartitions.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25036) Introduce stage-wised scheduling strategy

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25036:
--

 Summary: Introduce stage-wised scheduling strategy
 Key: FLINK-25036
 URL: https://issues.apache.org/jira/browse/FLINK-25036
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Lijie Wang


The scheduling of the adaptive batch job scheduler should be stage granularity, 
because the information for deciding parallelism can only be collected after 
the upstream stage is fully finished, so we need to introduce a new scheduling 
strategy: Stage-wised Scheduling Strategy.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] Drop Zookeeper 3.4

[Chesnay Schepler]

Hello,

I'd like to drop support for Zookeeper 3.4 in 1.15, upgrading the 
default to 3.5 with an opt-in for 3.6.


Supporting Zookeeper 3.4 (which is already EOL) prevents us from 
upgrading Curator to 5.x, which would allow us to properly fix an issue 
with inconsistent state. It is also required to eventually support ZK 3.6.

Re: [DISCUSS] Automated architectural tests

[Ingo Bürk]

Hi everyone,

I'm happy to announce now that we finished this test and it has been merged
(thanks to Chesnay for his help here). You can also find some documentation
for it here[1].
I haven't implemented all of the suggested rules, but we can of course
expand our checks here. The current rules show already quite a few
violations, some of which I will be going through in the next weeks. They
range from trivial fixes to things that need to be thought about.

[1]
https://github.com/apache/flink/blob/master/flink-architecture-tests/README.md


Ingo

On Wed, Sep 1, 2021 at 11:03 AM Ingo Bürk  wrote:

> Hello everyone,
>
> I would like to start a discussion on introducing automated tests for more
> architectural rather than stilistic topics. For example, here are a few
> things that seem worth checking to me (this is Table-API-focused since it
> is the subsystem I'm involved in):
>
> (a) All classes in o.a.f.table.api should be annotated with one
> of @Internal, @PublicEvolving, or @Public.
> (b) Classes whose name ends in *ConnectorOptions should be located in
> o.a.f.connector.*.table
> (c) Classes implementing DynamicSourceFactory / DynamicSinkFactory should
> have no static members of type ConfigOption
>
> There are probably significantly more cases worth checking, and also more
> involved ones (these are rather simple examples), like disallowing access
> between certain packages etc. There are two questions I would like to ask
> to the community:
>
> (1) Do you think such tests are useful in general?
> (2) What use cases come to mind for you?
>
> If the idea finds consensus, I would like to use (2) to investigate which
> tooling to use. An obvious candidate is Checkstyle, as this is already
> used. It also has the advantage of being well integrated in the IDE.
> However, it is limited to looking at single files only, and custom checks
> are pretty complicated and involved to implement[1]. Another possible tool
> is ArchUnit[2], which would be significantly easier to maintain and is more
> powerful, but in turn requires tests to be executed. If you have further
> suggestions (or thoughts) they would of course also be quite welcome,
> though for now I would focus on (1) and (2) and go from there to evaluate.
>
> [1] https://checkstyle.sourceforge.io/writingchecks.html
> [2] https://www.archunit.org/
>
>
> Best
> Ingo
>

[jira] [Created] (FLINK-25037) Compilation of compile_cron_python_wheels failed on AZP

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-25037:
-

 Summary: Compilation of compile_cron_python_wheels failed on AZP
 Key: FLINK-25037
 URL: https://issues.apache.org/jira/browse/FLINK-25037
 Project: Flink
  Issue Type: Bug
  Components: API / Python, Build System / Azure Pipelines
Affects Versions: 1.12.5
Reporter: Till Rohrmann
 Fix For: 1.12.6


The compilation of {{compile_cron_python_wheels}} failed on AZP with

{code}
==
Compiling Flink
==
Invoking mvn with 'mvn -Dmaven.wagon.http.pool=false --settings 
/__w/1/s/tools/ci/google-mirror-settings.xml 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS 
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
 --no-snapshot-updates -B -Dhadoop.version=2.8.3 -Dinclude_hadoop_aws 
-Dscala-2.11  clean deploy 
-DaltDeploymentRepository=validation_repository::default::file:/tmp/flink-validation-deployment
 -Dmaven.repo.local=/home/vsts/work/1/.m2/repository 
-Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2 
-Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -U -DskipTests'
[ERROR] Could not create local repository at /home/vsts/work/1/.m2/repository 
-> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/LocalRepositoryNotAccessibleException
==
Compiling Flink failed.
==
{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26968&view=logs&j=a29bcfe1-064d-50b9-354f-07802213a3c0&t=47ff6576-c9dc-5eab-9db8-183dcca3bede



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25038) Refactor FlinkContainer to split JM and TMs into different containers and supports HA

[Qingsheng Ren (Jira)]

Qingsheng Ren created FLINK-25038:
-

 Summary: Refactor FlinkContainer to split JM and TMs into 
different containers and supports HA
 Key: FLINK-25038
 URL: https://issues.apache.org/jira/browse/FLINK-25038
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Affects Versions: 1.15.0
Reporter: Qingsheng Ren
 Fix For: 1.15.0


Refactor FlinkContainer to let it run as a truly distributed system, and add HA 
support to enable JM failover



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Drop Zookeeper 3.4

[Matthias Pohl]

Thanks for starting this discussion, Chesnay. +1 from my side. It's time to
move forward with the ZK support considering the EOL of 3.4 you already
mentioned. The benefits we gain from upgrading Curator to 5.x as a
consequence is another plus point. Just for reference on the inconsistent
state issue you mentioned: FLINK-24543 [1].

Matthias

[1] https://issues.apache.org/jira/browse/FLINK-24543

On Wed, Nov 24, 2021 at 10:19 AM Chesnay Schepler 
wrote:

> Hello,
>
> I'd like to drop support for Zookeeper 3.4 in 1.15, upgrading the
> default to 3.5 with an opt-in for 3.6.
>
> Supporting Zookeeper 3.4 (which is already EOL) prevents us from
> upgrading Curator to 5.x, which would allow us to properly fix an issue
> with inconsistent state. It is also required to eventually support ZK 3.6.
>

[jira] [Created] (FLINK-25039) AZP fails with license check

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-25039:
-

 Summary: AZP fails with license check
 Key: FLINK-25039
 URL: https://issues.apache.org/jira/browse/FLINK-25039
 Project: Flink
  Issue Type: Bug
  Components: Build System / Azure Pipelines
Affects Versions: 1.15.0
Reporter: Till Rohrmann


The AZP build fails with a license check:

{code}
21:26:40,233 ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker
[] - Missing META-INF/LICENSE in 
/tmp/flink-validation-deployment/org/apache/flink/flink-sql-parquet_2.12/1.15-SNAPSHOT/flink-sql-parquet_2.12-1.15-20211123.212027-1-tests.jar
21:26:40,738 ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker
[] - The notice file in 
/tmp/flink-validation-deployment/org/apache/flink/flink-connector-cassandra_2.12/1.15-SNAPSHOT/flink-connector-cassandra_2.12-1.15-20211123.211736-1-tests.jar
 does not contain the expected entries.
21:26:40,739 ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker
[] - Missing META-INF/LICENSE in 
/tmp/flink-validation-deployment/org/apache/flink/flink-connector-cassandra_2.12/1.15-SNAPSHOT/flink-connector-cassandra_2.12-1.15-20211123.211736-1-tests.jar
21:26:41,673 ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker
[] - The notice file in 
/tmp/flink-validation-deployment/org/apache/flink/flink-kubernetes/1.15-SNAPSHOT/flink-kubernetes-1.15-20211123.212114-1-tests.jar
 does not contain the expected entries.
21:26:41,675 ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker
[] - Missing META-INF/LICENSE in 
/tmp/flink-validation-deployment/org/apache/flink/flink-kubernetes/1.15-SNAPSHOT/flink-kubernetes-1.15-20211123.212114-1-tests.jar
21:28:00,582 WARN  org.apache.flink.tools.ci.licensecheck.LicenseChecker
[] - Found a total of 5 severe license issues
==
License Check failed. See previous output for details.
==
##[error]Bash exited with code '1'.
{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26967&view=logs&j=946871de-358d-5815-3994-8175615bc253&t=e0240c62-4570-5d1c-51af-dd63d2093da1&l=30668
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26967&view=logs&j=e9d3d34f-3d15-59f4-0e3e-35067d100dfe&t=a7382ec4-87d2-5a9d-7c53-a2f93e317458&l=31863
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26967&view=logs&j=6e8542d7-de38-5a33-4aca-458d6c87066d&t=dffc2faa-5b48-5b4e-0797-dec1b1f74872&l=31863



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25040) FlinkKafkaInternalProducerITCase.testInitTransactionId failed on AZP

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-25040:
-

 Summary: FlinkKafkaInternalProducerITCase.testInitTransactionId 
failed on AZP
 Key: FLINK-25040
 URL: https://issues.apache.org/jira/browse/FLINK-25040
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.15.0
Reporter: Till Rohrmann
 Fix For: 1.15.0


The test {{FlinkKafkaInternalProducerITCase.testInitTransactionId}} failed on 
AZP with:

{code}
Nov 24 09:25:41 [ERROR] 
org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase.testInitTransactionId
  Time elapsed: 82.766 s  <<< ERROR!
Nov 24 09:25:41 org.apache.kafka.common.errors.TimeoutException: Timeout 
expired after 6 milliseconds while awaiting InitProducerId
Nov 24 09:25:41 
{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26987&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=15a22db7-8faa-5b34-3920-d33c9f0ca23c&l=6726



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25041) E2E tar ball cache fails without error message if target directory not specified

[Fabian Paul (Jira)]

Fabian Paul created FLINK-25041:
---

 Summary: E2E tar ball cache fails without error message if target 
directory not specified
 Key: FLINK-25041
 URL: https://issues.apache.org/jira/browse/FLINK-25041
 Project: Flink
  Issue Type: Bug
  Components: Test Infrastructure
Affects Versions: 1.13.3, 1.14.0, 1.15.0
Reporter: Fabian Paul


We want to verify if the variable has been set.
{code:java}
if [ -z "$E2E_TARBALL_CACHE" ] ; then
echo "You have to export the E2E Tarball Cache as E2E_TARBALL_CACHE"
exit 1
fi {code}
but the shown code immediately fails with an `unbound variable` error if the 
variable is not set and it does not evaluate the branch.

 

We should change it to something like this
{code:java}
if [ "${E2E_TARBALL_CACHE+x}" == x ] ; then
echo "You have to export the E2E Tarball Cache as E2E_TARBALL_CACHE" 
exit 1
fi  {code}
 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25042) Allow calls to @VisibleForTesting from inner classes

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-25042:


 Summary: Allow calls to @VisibleForTesting from inner classes
 Key: FLINK-25042
 URL: https://issues.apache.org/jira/browse/FLINK-25042
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.15.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.15.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25043) Allow calls to public @VisibleForTesting from the same package

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-25043:


 Summary: Allow calls to public @VisibleForTesting from the same 
package
 Key: FLINK-25043
 URL: https://issues.apache.org/jira/browse/FLINK-25043
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Chesnay Schepler
 Fix For: 1.15.0


Consider a class having some package-private method that is used by other 
classes in said package.

If this method is then needed from outside the package, and thus made public 
and annotated with VisibleForTesting, then the architecture tests currently 
flag the original usage as well.

We could think about allowing package-private access if the method is public.

On the other hand, if the method was originally annotated with 
VisibleForTesting, then marking it as public would remove a violation, which 
would be incorrect.

Maybe we need to extend our VisibleForTesting annotation to provide this 
information.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

[Jingsong Li]

Hi Stephan,

Thanks for your reply.

Data never expires automatically.

If there is a need for data retention, the user can choose one of the
following options:
- In the SQL for querying the managed table, users filter the data by themselves
- Define the time partition, and users can delete the expired
partition by themselves. (DROP PARTITION ...)
- In the future version, we will support the "DELETE FROM" statement,
users can delete the expired data according to the conditions.

So to answer your question:

> Will the VMQ send retractions so that the data will be removed from the table 
> (via compactions)?

The current implementation is not sending retraction, which I think
theoretically should be sent, currently the user can filter by
subsequent conditions.
And yes, the subscriber would not see strictly a correct result. I
think this is something we can improve for Flink SQL.

> Do we want time retention semantics handled by the compaction?

Currently, no, Data never expires automatically.

> Do we want to declare those types of queries "out of scope" initially?

I think we want users to be able to use three options above to
accomplish their requirements.

I will update FLIP to make the definition clearer and more explicit.

Best,
Jingsong

On Wed, Nov 24, 2021 at 5:01 AM Stephan Ewen  wrote:
>
> Thanks for digging into this.
> Regarding this query:
>
> INSERT INTO the_table
>   SELECT window_end, COUNT(*)
> FROM (TUMBLE(TABLE interactions, DESCRIPTOR(ts), INTERVAL '5' MINUTES))
> GROUP BY window_end
>   HAVING now() - window_end <= INTERVAL '14' DAYS;
>
> I am not sure I understand what the conclusion is on the data retention 
> question, where the continuous streaming SQL query has retention semantics. I 
> think we would need to answer the following questions (I will call the query 
> that computed the managed table the "view materializer query" - VMQ).
>
> (1) I guess the VMQ will send no updates for windows beyond the "retention 
> period" is over (14 days), as you said. That makes sense.
>
> (2) Will the VMQ send retractions so that the data will be removed from the 
> table (via compactions)?
>   - if yes, this seems semantically better for users, but it will be 
> expensive to keep the timers for retractions.
>   - if not, we can still solve this by adding filters to queries against the 
> managed table, as long as these queries are in Flink.
>   - any subscriber to the changelog stream would not see strictly a correct 
> result if we are not doing the retractions
>
> (3) Do we want time retention semantics handled by the compaction?
>   - if we say that we lazily apply the deletes in the queries that read the 
> managed tables, then we could also age out the old data during compaction.
>   - that is cheap, but it might be too much of a special case to be very 
> relevant here.
>
> (4) Do we want to declare those types of queries "out of scope" initially?
>   - if yes, how many users are we affecting? (I guess probably not many, but 
> would be good to hear some thoughts from others on this)
>   - should we simply reject such queries in the optimizer as "not possible to 
> support in managed tables"? I would suggest that, always better to tell users 
> exactly what works and what not, rather than letting them be surprised in the 
> end. Users can still remove the HAVING clause if they want the query to run, 
> and that would be better than if the VMQ just silently ignores those 
> semantics.
>
> Thanks,
> Stephan
>


-- 
Best, Jingsong Lee

[jira] [Created] (FLINK-25044) Add More Unit Test For Pulsar Source

[Yufei Zhang (Jira)]

Yufei Zhang created FLINK-25044:
---

 Summary: Add More Unit Test For Pulsar Source
 Key: FLINK-25044
 URL: https://issues.apache.org/jira/browse/FLINK-25044
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Pulsar
Reporter: Yufei Zhang


We should enhance the pulsar source connector tests by adding more unit tests.

 
 * SourceReader
 * SplitReader
 * Enumerator
 * SourceBuilder



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25045) Introduce AdaptiveBatchScheduler

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25045:
--

 Summary: Introduce AdaptiveBatchScheduler
 Key: FLINK-25045
 URL: https://issues.apache.org/jira/browse/FLINK-25045
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Lijie Wang


Introduce AdaptiveBatchScheduler



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25046) Convert forward edge to rescale when using adapive batch scheduler

[Lijie Wang (Jira)]

Lijie Wang created FLINK-25046:
--

 Summary: Convert forward edge to rescale when using adapive batch 
scheduler
 Key: FLINK-25046
 URL: https://issues.apache.org/jira/browse/FLINK-25046
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Lijie Wang






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25047) Resolve most (trivial) architectural violations in flink-table

[Jira]

Ingo Bürk created FLINK-25047:
-

 Summary: Resolve most (trivial) architectural violations in 
flink-table
 Key: FLINK-25047
 URL: https://issues.apache.org/jira/browse/FLINK-25047
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: Ingo Bürk
Assignee: Ingo Bürk






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25048) 在某个option后设置watermark，如果这个option里面有 outputTag，则此outputTag 不会有数据输出，只会在主流输出数据

[xinli liang (Jira)]

xinli liang created FLINK-25048:
---

 Summary: 在某个option后设置watermark，如果这个option里面有 outputTag，则此outputTag 
不会有数据输出，只会在主流输出数据
 Key: FLINK-25048
 URL: https://issues.apache.org/jira/browse/FLINK-25048
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Reporter: xinli liang
 Attachments: 1637770386(1).jpg, 1637770540(1).jpg, test.java

// 1. 创建流式执行环境
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
// 2. 读取文件
DataStreamSource lineDSS = env.socketTextStream("hadoop102", );

OutputTag outputTag = new OutputTag("test"){};
// 3. 转换数据格式

SingleOutputStreamOperator process = lineDSS.process(new 
ProcessFunction() {
@Override
public void processElement(String value, Context ctx, Collector out) 
throws Exception {
String[] s = value.split(" ");
String word = s[0];
String ts = s[1];
if (word.startsWith("a")) {
out.collect(value);
} else {
ctx.output(outputTag, value);
}
}
})
.assignTimestampsAndWatermarks(WatermarkStrategy
.forBoundedOutOfOrderness(Duration.ofSeconds(4))
.withTimestampAssigner((data,ts)-> Long.parseLong(data.split(" ")[1])))
;

process.print("主流>>>");
process.getSideOutput(outputTag).print("侧输出流>>>");


// 4. 执行
try {
env.execute();
} catch (Exception e) {
e.printStackTrace();
}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] FLIP-188 Introduce Built-in Dynamic Table Storage

[Stephan Ewen]

Thanks for all the details and explanation.

With the conclusion of the discussion, also +1 from my side for this FLIP

On Sat, Nov 13, 2021 at 12:23 PM Jingsong Li  wrote:

> Thanks Stephan and Timo, I have a rough look at your replies. They are
> all valuable opinions. I will take time to discuss, explain and
> improve them.
>
> Hi Timo,
> > At least a final "I will start the vote soon. Last call for comments."
> would have been nice.
>
> I replied in the DISCUSS thread that we began to vote. If there are
> supplementary comments or reply "pause voting first, I will reply
> later", we can suspend or cancel the voting at any time.
> I understand why the FLIP must take three days to vote, so that more
> people can see it and put forward their opinions.
>
> Best,
> Jingsong
>
> On Sat, Nov 13, 2021 at 1:27 AM Timo Walther  wrote:
> >
> > Hi everyone,
> >
> > even though the DISCUSS thread was open for 2 weeks. I have the feeling
> > that the VOTE was initiated to quickly. At least a final "I will start
> > the vote soon. Last call for comments." would have been nice.
> >
> > I also added some comments in the DISCUSS thread. Let's hope we can
> > resolve those soon.
> >
> > Regards,
> > Timo
> >
> > On 12.11.21 16:36, Stephan Ewen wrote:
> > > Hi all!
> > >
> > > I have a few questions on the design still, posted those in the
> [DISCUSS]
> > > thread.
> > > It would be great to clarify those first before concluding this vote.
> > >
> > > Thanks,
> > > Stephan
> > >
> > >
> > > On Fri, Nov 12, 2021 at 7:22 AM Jark Wu  wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Thanks for the great work Jingsong!
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Thu, 11 Nov 2021 at 19:41, JING ZHANG 
> wrote:
> > >>
> > >>> +1 (non-binding)
> > >>>
> > >>> A small suggestion:
> > >>> The message queue is currently used to store middle layer data of the
> > >>> streaming data warehouse. We hope use built-in dynamic table storage
> to
> > >>> store those middle layer.
> > >>> But those middle data of the streaming data warehouse are often
> provided
> > >> to
> > >>> all business teams in a company. Some teams have not use Apache
> Flink as
> > >>> compute engine yet. In order to continue server those teams, the
> data in
> > >>> built-in dynamic table storage may be needed to copied to message
> queue
> > >>> again.
> > >>> If *the built-in storage could provide same consumer API as the
> commonly
> > >>> used message queues*, data copying may be avoided. So the built-in
> > >> dynamic
> > >>> table storage may be promoted faster in the streaming data warehouse
> > >>> business.
> > >>>
> > >>> Best regards,
> > >>> Jing Zhang
> > >>>
> > >>> Yufei Zhang  于2021年11月11日周四 上午9:34写道：
> > >>>
> >  Hi,
> > 
> >  +1 (non-binding)
> > 
> >  Very interesting design. I saw a lot of discussion on the generic
> >  interface design, good to know it will address extensibility.
> > 
> >  Cheers,
> >  Yufei
> > 
> > 
> >  On 2021/11/10 02:51:55 Jingsong Li wrote:
> > > Hi everyone,
> > >
> > > Thanks for all the feedback so far. Based on the discussion[1] we
> > >> seem
> > > to have consensus, so I would like to start a vote on FLIP-188 for
> > > which the FLIP has now also been updated[2].
> > >
> > > The vote will last for at least 72 hours (Nov 13th 3:00 GMT) unless
> > > there is an objection or insufficient votes.
> > >
> > > [1]
> https://lists.apache.org/thread/tqyn1cro5ohl3c3fkjb1zvxbo03sofn7
> > > [2]
> > 
> > >>>
> > >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage
> > >
> > > Best,
> > > Jingsong
> > >
> > 
> > >>>
> > >>
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

[DISCUSS] Releasing Flink 1.14.1

[Martijn Visser]

Hi all,

I would like to start a discussion on releasing Flink 1.14.1. Flink 1.14
was released on the 29th of September [1] and so far 107 issues have been
resolved, including multiple blockers and critical priorities [2].

There are currently 169 open tickets which contain a fixVersion for 1.14.1
[3]. I'm including the ones that are currently marked as critical or a
blocker to verify if these should be included in Flink 1.14.1. It would be
great if those that are assigned or working on one or more of these tickets
can give an update on its status.

* https://issues.apache.org/jira/browse/FLINK-24543 - Zookeeper connection
issue causes inconsistent state in Flink -> I think this depends on the
outcome of dropping Zookeeper 3.4 as was proposed on the Dev mailing list
* https://issues.apache.org/jira/browse/FLINK-25027 - Allow GC of a
finished job's JobMaster before the slot timeout is reached
* https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader leak with
ThreadLocals on the JM when submitting a job through the REST API
* https://issues.apache.org/jira/browse/FLINK-24789 - IllegalStateException
with CheckpointCleaner being closed already
* https://issues.apache.org/jira/browse/FLINK-24328 - Long term fix for
receiving new buffer size before network reader configured -> I'm not sure
if this would end up in Flink 1.14.1, I think it's more likely that it
would be Flink 1.15. Anton/Dawid, could you confirm this?
* https://issues.apache.org/jira/browse/FLINK-23946 - Application mode
fails fatally when being shut down -> This depends on
https://issues.apache.org/jira/browse/FLINK-24038 and I don't see much
happening there, so I also expect that this would move to Flink 1.15.
David, could you confirm?
* https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey constraint
is lost with multiple sources join in SQL
* https://issues.apache.org/jira/browse/FLINK-21788 - Throw
PartitionNotFoundException if the partition file has been lost for blocking
shuffle -> I'm also expecting that this would move to Flink 1.15, can you
confirm Yingjie ?

There are quite some other tickets that I've excluded from this list,
because they are either test instabilities or are not depending on a Flink
release to be resolved.

Note: there are quite a few test instabilities in the list and help on
those is always appreciated. You can check all unassigned tickets
instabilities in Jira [4].

Are there any other open tickets that we should wait for? Is there a PMC
member who would like to manage the release? I'm more than happy to help
with monitoring the status of the tickets.

Best regards,

Martijn

[1] https://flink.apache.org/news/2021/09/29/release-1.14.0.html
[2]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
[3]
https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC

[4]
https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20AND%20labels%20%3D%20test-stability%20AND%20assignee%20in%20(EMPTY)%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC

Martijn Visser | Product Manager

mart...@ververica.com




Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

Re: [DISCUSS] Releasing Flink 1.14.1

[Yingjie Cao]

Hi Martijn,

I moved the fix version of "FLINK-21788
 - Throw
PartitionNotFoundException if the partition file has been lost for blocking
shuffle" to 1.15.0

Best,
Yingjie

Martijn Visser  于2021年11月25日周四 上午2:40写道：

> Hi all,
>
> I would like to start a discussion on releasing Flink 1.14.1. Flink 1.14
> was released on the 29th of September [1] and so far 107 issues have been
> resolved, including multiple blockers and critical priorities [2].
>
> There are currently 169 open tickets which contain a fixVersion for 1.14.1
> [3]. I'm including the ones that are currently marked as critical or a
> blocker to verify if these should be included in Flink 1.14.1. It would be
> great if those that are assigned or working on one or more of these tickets
> can give an update on its status.
>
> * https://issues.apache.org/jira/browse/FLINK-24543 - Zookeeper connection
> issue causes inconsistent state in Flink -> I think this depends on the
> outcome of dropping Zookeeper 3.4 as was proposed on the Dev mailing list
> * https://issues.apache.org/jira/browse/FLINK-25027 - Allow GC of a
> finished job's JobMaster before the slot timeout is reached
> * https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader leak
> with
> ThreadLocals on the JM when submitting a job through the REST API
> * https://issues.apache.org/jira/browse/FLINK-24789 -
> IllegalStateException
> with CheckpointCleaner being closed already
> * https://issues.apache.org/jira/browse/FLINK-24328 - Long term fix for
> receiving new buffer size before network reader configured -> I'm not sure
> if this would end up in Flink 1.14.1, I think it's more likely that it
> would be Flink 1.15. Anton/Dawid, could you confirm this?
> * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode
> fails fatally when being shut down -> This depends on
> https://issues.apache.org/jira/browse/FLINK-24038 and I don't see much
> happening there, so I also expect that this would move to Flink 1.15.
> David, could you confirm?
> * https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey constraint
> is lost with multiple sources join in SQL
> * https://issues.apache.org/jira/browse/FLINK-21788 - Throw
> PartitionNotFoundException if the partition file has been lost for blocking
> shuffle -> I'm also expecting that this would move to Flink 1.15, can you
> confirm Yingjie ?
>
> There are quite some other tickets that I've excluded from this list,
> because they are either test instabilities or are not depending on a Flink
> release to be resolved.
>
> Note: there are quite a few test instabilities in the list and help on
> those is always appreciated. You can check all unassigned tickets
> instabilities in Jira [4].
>
> Are there any other open tickets that we should wait for? Is there a PMC
> member who would like to manage the release? I'm more than happy to help
> with monitoring the status of the tickets.
>
> Best regards,
>
> Martijn
>
> [1] https://flink.apache.org/news/2021/09/29/release-1.14.0.html
> [2]
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
> [3]
>
> https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
>
> [4]
>
> https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20AND%20labels%20%3D%20test-stability%20AND%20assignee%20in%20(EMPTY)%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
>
> Martijn Visser | Product Manager
>
> mart...@ververica.com
>
> 
>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward  - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>

[jira] [Created] (FLINK-25049) sql-client.sh execute batch job in async mode failed with java.io.FileNotFoundException

[macdoor615 (Jira)]

macdoor615 created FLINK-25049:
--

 Summary: sql-client.sh execute batch job in async mode failed with 
java.io.FileNotFoundException
 Key: FLINK-25049
 URL: https://issues.apache.org/jira/browse/FLINK-25049
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.14.0
 Environment: {code:java}
//代码占位符
{code}
Reporter: macdoor615


execute multi simple sql in a sql file, like 

 
{code:java}
insert overwrite bnpmp.p_biz_hcd_5m select * from bnpmp.p_biz_hcd_5m where 
dt='2021-11-22';
insert overwrite bnpmp.p_biz_hjr_5m select * from bnpmp.p_biz_hjr_5m where 
dt='2021-11-22';
insert overwrite bnpmp.p_biz_hswtv_5m select * from bnpmp.p_biz_hswtv_5m where 
dt='2021-11-22';
...
{code}
 

if 
{code:java}
SET table.dml-sync = true;{code}
execute properly.

if
{code:java}
SET table.dml-sync = false;{code}
some SQL Job failed with the following error
 
{code:java}
 Caused by: java.lang.Exception: Failed to finalize execution on master ... 37 
more Caused by: org.apache.flink.table.api.TableException: Exception in 
finalizeGlobal at 
org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:91)
 at 
org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:148)
 at 
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.vertexFinished(DefaultExecutionGraph.java:1086)
 ... 36 more Caused by: java.io.FileNotFoundException: File 
hdfs://service1/user/hive/warehouse/bnpmp.db/p_snmp_f5_traffic_5m/.staging_1637821441497
 does not exist. at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:901)
 at 
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:112)
 at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961)
 at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:958)
 at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:958)
 at 
org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:170)
 at 
org.apache.flink.table.filesystem.PartitionTempFileManager.listTaskTemporaryPaths(PartitionTempFileManager.java:104)
 at 
org.apache.flink.table.filesystem.FileSystemCommitter.commitPartitions(FileSystemCommitter.java:77)
 at 
org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:89)
 ... 38 more
 
{code}
 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25050) Translate "Metrics" page of "Operations" into Chinese

[ZhiJie Yang (Jira)]

ZhiJie Yang created FLINK-25050:
---

 Summary: Translate "Metrics" page of "Operations" into Chinese
 Key: FLINK-25050
 URL: https://issues.apache.org/jira/browse/FLINK-25050
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation
Reporter: ZhiJie Yang


The page url is 
https://nightlies.apache.org/flink/flink-docs-master/docs/zh/docs/ops/metrics/

The markdown file is located in flink/docs/content.zh/docs/ops/metrics.md



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

RocksDBMapState get the binary key bytes

[Zen4YYDS]

Hi devs:

 Using RocksDB, when key and namespace both have variable binary length, to 
prevent [key, namespace] have equal binary number, we add key length and 
namespace length after key and namespace respectively. Then the format is:
Keygroup – key -keyLength– namespace-namespaceLenth

 Then what about we use a fixed length key and variable length namespace 
and userkey. In current implement, I found the binary key format is as below:

   Keygroup – key – namespace- userkey

Think about following situation, I think we may get the same value for 
different [namespace, userkey]. or I get something wrong?

Keygroup  key   namespace  userkey
1 1  11  1
　1  1 111


从 Windows 版邮件发送

Re: [DISCUSS] Releasing Flink 1.14.1

[Fabian Paul]

Hi Martijn,

Thanks for bringing up this topic. I think it would be great to release a
patch version of 1.14 before the end of the year.

Currently, FLINK-24596  is
in progress and I would block the release until it is merged because it
unblocks several use cases by our users. I think the ticket is done by the
end of this week.

Best,
Fabian


On Thu, Nov 25, 2021 at 3:31 AM Yingjie Cao  wrote:

> Hi Martijn,
>
> I moved the fix version of "FLINK-21788
>  - Throw
> PartitionNotFoundException if the partition file has been lost for blocking
> shuffle" to 1.15.0
>
> Best,
> Yingjie
>
> Martijn Visser  于2021年11月25日周四 上午2:40写道：
>
> > Hi all,
> >
> > I would like to start a discussion on releasing Flink 1.14.1. Flink 1.14
> > was released on the 29th of September [1] and so far 107 issues have been
> > resolved, including multiple blockers and critical priorities [2].
> >
> > There are currently 169 open tickets which contain a fixVersion for
> 1.14.1
> > [3]. I'm including the ones that are currently marked as critical or a
> > blocker to verify if these should be included in Flink 1.14.1. It would
> be
> > great if those that are assigned or working on one or more of these
> tickets
> > can give an update on its status.
> >
> > * https://issues.apache.org/jira/browse/FLINK-24543 - Zookeeper
> connection
> > issue causes inconsistent state in Flink -> I think this depends on the
> > outcome of dropping Zookeeper 3.4 as was proposed on the Dev mailing list
> > * https://issues.apache.org/jira/browse/FLINK-25027 - Allow GC of a
> > finished job's JobMaster before the slot timeout is reached
> > * https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader leak
> > with
> > ThreadLocals on the JM when submitting a job through the REST API
> > * https://issues.apache.org/jira/browse/FLINK-24789 -
> > IllegalStateException
> > with CheckpointCleaner being closed already
> > * https://issues.apache.org/jira/browse/FLINK-24328 - Long term fix for
> > receiving new buffer size before network reader configured -> I'm not
> sure
> > if this would end up in Flink 1.14.1, I think it's more likely that it
> > would be Flink 1.15. Anton/Dawid, could you confirm this?
> > * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode
> > fails fatally when being shut down -> This depends on
> > https://issues.apache.org/jira/browse/FLINK-24038 and I don't see much
> > happening there, so I also expect that this would move to Flink 1.15.
> > David, could you confirm?
> > * https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey
> constraint
> > is lost with multiple sources join in SQL
> > * https://issues.apache.org/jira/browse/FLINK-21788 - Throw
> > PartitionNotFoundException if the partition file has been lost for
> blocking
> > shuffle -> I'm also expecting that this would move to Flink 1.15, can you
> > confirm Yingjie ?
> >
> > There are quite some other tickets that I've excluded from this list,
> > because they are either test instabilities or are not depending on a
> Flink
> > release to be resolved.
> >
> > Note: there are quite a few test instabilities in the list and help on
> > those is always appreciated. You can check all unassigned tickets
> > instabilities in Jira [4].
> >
> > Are there any other open tickets that we should wait for? Is there a PMC
> > member who would like to manage the release? I'm more than happy to help
> > with monitoring the status of the tickets.
> >
> > Best regards,
> >
> > Martijn
> >
> > [1] https://flink.apache.org/news/2021/09/29/release-1.14.0.html
> > [2]
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
> > [3]
> >
> >
> https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
> >
> > [4]
> >
> >
> https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20AND%20labels%20%3D%20test-stability%20AND%20assignee%20in%20(EMPTY)%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
> >
> > Martijn Visser | Product Manager
> >
> > mart...@ververica.com
> >
> > 
> >
> >
> > Follow us @VervericaData
> >
> > --
> >
> > Join Flink Forward  - The Apache Flink
> > Conference
> >
> > Stream Processing | Event Driven | Real Time
> >
>