[jira] [Created] (FLINK-29728) TablePlanner prevents Flink from starting is working directory is a symbolic link

[Angelo Kastroulis (Jira)]

Angelo Kastroulis created FLINK-29728:
-

 Summary: TablePlanner prevents Flink from starting is working 
directory is a symbolic link
 Key: FLINK-29728
 URL: https://issues.apache.org/jira/browse/FLINK-29728
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.2
Reporter: Angelo Kastroulis


The Flink runtime throws an exception when using the table API if the working 
directory is a symbolic link. This is the case when run on AWS EMR with Yarn. 
There is a similar issue 
[here|https://issues.apache.org/jira/browse/FLINK-20267] and I believe the same 
fix applied there would work.

 

 
{code:java}
Caused by: org.apache.flink.table.api.TableException: Could not initialize the 
table planner components loader.
    at 
org.apache.flink.table.planner.loader.PlannerModule.(PlannerModule.java:123)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.planner.loader.PlannerModule.(PlannerModule.java:52)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.planner.loader.PlannerModule$PlannerComponentsHolder.(PlannerModule.java:131)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.planner.loader.PlannerModule.getInstance(PlannerModule.java:135)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.planner.loader.DelegateExecutorFactory.(DelegateExecutorFactory.java:34)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
~[?:1.8.0_342]
    at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 ~[?:1.8.0_342]
    at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 ~[?:1.8.0_342]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 
~[?:1.8.0_342]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_342]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) 
~[?:1.8.0_342]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) 
~[?:1.8.0_342]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_342]
    at 
org.apache.flink.table.factories.ServiceLoaderUtil.load(ServiceLoaderUtil.java:42)
 ~[flink-table-api-java-uber-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:798)
 ~[flink-table-api-java-uber-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:517)
 ~[flink-table-api-java-uber-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.create(TableEnvironmentImpl.java:276)
 ~[flink-table-api-java-uber-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.api.TableEnvironment.create(TableEnvironment.java:93) 
~[flink-table-api-java-uber-1.15.1.jar:1.15.1]
    at com.ballista.Hermes.BCSE$.useLocalCatalog(BCSE.scala:210) ~[?:?]
    at com.ballista.Hermes.BCSE$.main(BCSE.scala:114) ~[?:?]
    at com.ballista.Hermes.BCSE.main(BCSE.scala) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_342]
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_342]
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_342]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_342]
    at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
 ~[flink-dist-1.15.1.jar:1.15.1]
    ... 7 more
Caused by: java.nio.file.FileAlreadyExistsException: /tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) 
~[?:1.8.0_342]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) 
~[?:1.8.0_342]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) 
~[?:1.8.0_342]
    at 
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
 ~[?:1.8.0_342]
    at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_342]
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) 
~[?:1.8.0_342]
    at java.nio.file.Files.createDirectories(Files.java:727) ~[?:1.8.0_342]
    at 
org.apache.flink.table.planner.loader.PlannerModule.(PlannerModule.java:96)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.planner.loader.PlannerModule.(PlannerModule.java:52)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.planner.loader.PlannerModule$PlannerComponentsHolder.(PlannerModule.java:131)
 ~[flink-table-planner-loader-1.15.1.jar:1.15.1]
    at 
org.apache.flink.table.planner.loader.PlannerModule.getInstance(PlannerModule.java:135)
 ~[flink-table-planner-loader-1.15

Re: [DISCUSS] REST API to suspend & resume checkpointing

[Saurabh Kaul]

Hi Piotr,

Sorry for the confusion. I'm referring to latency concerns at 2 points -

1. In producing a full snapshot, I see this is noted as follows in the Flip
[1]:
> This means that if the underlying FileSystem doesn’t support fast
duplication, incremental savepoints will most likely be still slower
compared to incremental checkpoints.
In our use case for example, we use S3 file system so we have some
additional overhead of duplication.
2. I mixed up the savepoint recovery modes, I think the default NO_CLAIM
mode would be more appropriate for production use generally since CLAIM
mode carries the risk of accidental savepoint deletion when future
checkpoints can build on top of it. Manual checkpoint deletion is not a
concern. NO_CLAIM would require the first successful checkpoint to be a
full checkpoint.

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-203

On Fri, Oct 21, 2022 at 1:31 AM Piotr Nowojski  wrote:

> Hi Saurabh,
>
> > 1. Stop with native savepoint does solve any races and produces a
> > predictable restoration point, but producing a self-contained snapshot
> and
> > using CLAIM mode in re-running is not necessary here and adds latency.
>
> Why do you think it adds latency? CLAIM mode is actually the fastest one
> with zero additional overheads. Native savepoints are almost an equivalent
> of checkpoints.
>
> Best,
> Piotrek
>
>
> pt., 21 paź 2022 o 00:51 Saurabh Kaul  napisał(a):
>
> > Hi, thanks for the quick responses,
> >
> > I think a stop-with-checkpoint idea is overlapping well with the
> > requirements.
> >
> > 1. Stop with native savepoint does solve any races and produces a
> > predictable restoration point, but producing a self-contained snapshot
> and
> > using CLAIM mode in re-running is not necessary here and adds latency.
> > Stop-with-checkpoint doesn't have these issues. It adds some downtime in
> > waiting for a checkpoint to be completed but reduces replay time in the
> new
> > cluster which is a good trade-off. Since in this scenario of job
> migration
> > the job and/or job configuration is not changing; it should ideally be as
> > fast as a regular failover scenario (like a TM going down).
> > 2. Taking complete ownership of triggering checkpoints and making them
> more
> > configurable could be feasible but are less effective comparatively in
> > terms of stopping the job for the primary purpose of low-downtime
> migration
> > of the job. Stop-with-checkpoint solves it more directly.
> >
> > Looking forward to hearing thoughts on this.
> >
> > On Thu, Oct 20, 2022 at 3:31 AM Piotr Nowojski 
> > wrote:
> >
> > > Hi Saurabh,
> > >
> > > Thanks for reaching out with the proposal. I have some mixed feelings
> > about
> > > this for a couple of reasons:
> > >
> > > 1. It sounds like the core problem that you are describing is the race
> > > condition between shutting down the cluster and completion of new
> > > checkpoints. My first thought would be as Jing's, why don't you use
> > > stop-with-savepoint? Especially the native savepoint? You can recover
> > from
> > > it using --claim mode, so the whole process should be quite fast
> > actually.
> > > 2. The same issue, not knowing the latest completed checkpoint id,
> > plagued
> > > us with some internal tests for quite a bit, so maybe this would also
> be
> > > worth considering to address instead? Like leaving in some text file
> the
> > > last completed checkpoint id? Or providing a way to read this from some
> > > existing metadata files? However in our tests we actually fixed/worked
> > > around that with manually triggering of checkpoints. The predecessor of
> > > FLINK-27101 [1], FLINK-24280 [2], was implemented to address this exact
> > > issue.  Which brings me to...
> > > 3. You could actually just use the REST API to trigger all checkpoints
> > > manually. The idea behind FLINK-27101 [1] was to add full flexibility
> to
> > > the users, without adding much complexity to the system. If we start
> > adding
> > > more REST calls to control checkpointing behaviour it would complicate
> > the
> > > system.
> > > 4. If at all, I would think more towards a more generic idea of
> > dynamically
> > > reconfiguring the system. We could provide a generic way to dynamically
> > > change configuration options. We wouldn't be able to support all
> > > configurations, and furthermore, each "dynamic" option would have to be
> > > handled/passed down to and through the system differently, BUT we
> > wouldn't
> > > have to do all of that at once. We could start with a very limited set
> of
> > > dynamic options, for example just with the checkpointing interval. This
> > > must have been considered/discussed before, so I might be missing lots
> of
> > > things.
> > > 5. Another direction, if 1. is not an option for some reason, is to
> > provide
> > > a stop-with-checkpoint feature?
> > >
> > > Best Piotrek
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-27101
> > > [2] https://issues.apache.org/jira/browse/FL

Re: [DISCUSS] Create a dedicated aws-base connector repository

[Danny Cranmer]

Thanks Chesnay for the suggestion, I will investigate this option.

Related to the single repo idea, I have considered it in the past. Are you
proposing we also use a single version between all connectors? If we have a
single version then it makes sense to combine them in a single repo, if
they are separate versions, then splitting them makes sense. This was
discussed last year more generally [1] and the consensus was "we ultimately
propose to have a single repository per connector".

Combining all AWS connectors into a single repo with a single version is
inline with how the AWS SDK works, therefore AWS users are familiar with
this approach. However it is frustrating that we would have to release all
connectors to fix a bug or add a feature in one of them. Example: a user is
using Kinesis Data Streams only (the most popular and mature connector),
and we evolve the version from 1.x to 2.y (or 1.x to 1.y) for a DynamoDB
change.

I am torn and will think some more, but it would be great to hear other
people's opinions.

[1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm

Thanks,
Danny

On Fri, Oct 21, 2022 at 3:11 PM Jing Ge  wrote:

> I agree with Jark. It would be easier for the further development and
> maintenance, if all aws related connectors and the base module are in the
> same repo. It might make sense to upgrade the flink-connector-dynamodb to
> flink-connector-aws and move the other modules including the
> flink-connector-aws-base into it. The aws sdk could be managed in
> flink-connector-aws-base. Any future common connector features could also
> be developed in the base module.
>
> Best regards,
> Jing
>
> On Fri, Oct 21, 2022 at 1:26 PM Jark Wu  wrote:
>
>> How about creating a new repository flink-connector-aws and merging
>> dynamodb, kinesis firehouse into it?
>> This can reduce the maintenance for complex dependencies and make the
>> release easy.
>> I think the maintainers of aws-releated connectors are the same people.
>>
>> Best,
>> Jark
>>
>> > 2022年10月21日 17:41，Chesnay Schepler  写道：
>> >
>> > I would not go with 2); I think it'd just be messy .
>> >
>> > Here's another option:
>> >
>> > Create another repository (aws-connector-base) (following the
>> externalization model), add it as a sub-module to the downstream
>> repositories, and make it part of the release process of said connector.
>> >
>> > I.e., we never create a release for aws-connector-bose, but release it
>> as part of the connector.
>> > This main benefit here is that we'd always be able to make changes to
>> the aws-base code without delaying connector releases.
>> > I would assume that any added overhead due to _technically_ releasing
>> the aws code multiple times to be negligible.
>> >
>> >
>> > On 20/10/2022 22:38, Danny Cranmer wrote:
>> >> Hello all,
>> >>
>> >> Currently we have 2 AWS Flink connectors in the main Flink codebase
>> >> (Kinesis Data Streams and Kinesis Data Firehose) and one new
>> externalized
>> >> connector in progress (DynamoDB). Currently all three of these use
>> common
>> >> AWS utilities from the flink-connector-aws-base module. Common code
>> >> includes client builders, property keys, validation, utils etc.
>> >>
>> >> Once we externalize the connectors, leaving flink-connector-aws-base
>> in the
>> >> main Flink repository will restrict our ability to evolve the
>> connectors
>> >> quickly. For example, as part of the DynamoDB connector build we are
>> >> considering adding a general retry strategy config that can be
>> leveraged by
>> >> all connectors. We would need to block on Flink 1.17 for this.
>> >>
>> >> In the past we have tried to keep the AWS SDK version consistent across
>> >> connectors, with the externalization this is more likely to diverge.
>> >>
>> >> Option 1: I propose we create a new repository, flink-connector-aws,
>> which
>> >> we can move the flink-connector-aws-base module to and create a new
>> >> flink-connector-aws-parent to manage SDK versions. Each of the
>> externalized
>> >> AWS connectors will depend on this new module and parent. Downside is
>> an
>> >> additional module to release per Flink version, however I will
>> volunteer to
>> >> manage this.
>> >>
>> >> Option 2: We can move the flink-connector-aws-base module and create
>> >> flink-connector-parent within the flink-connector-shared-utils repo [2]
>> >>
>> >> Option 3: We do nothing.
>> >>
>> >> For option 1+2 we will follow the general externalized connector
>> versioning
>> >> strategy and rules.
>> >>
>> >> I am inclined towards option 1, and appreciate feedback from the
>> community.
>> >>
>> >> [1]
>> >>
>> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
>> >> [2] https://github.com/apache/flink-connector-shared-utils
>> >>
>> >> Thanks,
>> >> Danny
>> >>
>> >
>>
>>

Re: [Cassandra] source connector

[Ryan Skraba]

There's definitely consensus on externalizing the flink connectors!  I've
been tracking the progress and I'd be happy to provide support on Cassandra
if you'd like.

There's some new information at
https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development

The first step to externalizing Cassandra would be for a PMC member to
create the flink-connector-cassandra repository for the project.

If I understand correctly, this shouldn't block your PR, since the core
connector (inside apache/flink) and the external one
(apache/flink-cassandra-connector) should be kept in sync for one major
release cycle.  For my own benefit, I can start by reviewing your PR!

All my best, Ryan

On Fri, Oct 21, 2022 at 11:51 AM Etienne Chauchot 
wrote:

> Hi,
>
> Yes sure, if there is a consensus on moving, just tell me where to move
> my PR to.
>
> Best
>
> Etienne
>
> Le 19/10/2022 à 18:20, Alexander Fedulov a écrit :
> > Hi Etienne,
> >
> > thanks for your contribution. In light of the current efforts to
> > externalize connectors, do you think we could maybe combine the new
> > implementation with moving it into an external repository instead of
> > merging into Flink main?
> >
> > Best,
> > Alexander Fedulov
> >
> > On Fri, Oct 14, 2022 at 4:18 PM Etienne Chauchot 
> > wrote:
> >
> >> Hi all,
> >>
> >> As promised, I have developed the Cassandra source connector based on
> >> FLIP-27. I've just submitted the
> >> PR:https://github.com/apache/flink/pull/21073
> >>
> >>
> >> Best
> >>
> >> Etienne
> >>
>

Re: [DISCUSS] Create a dedicated aws-base connector repository

[Jing Ge]

I agree with Jark. It would be easier for the further development and
maintenance, if all aws related connectors and the base module are in the
same repo. It might make sense to upgrade the flink-connector-dynamodb to
flink-connector-aws and move the other modules including the
flink-connector-aws-base into it. The aws sdk could be managed in
flink-connector-aws-base. Any future common connector features could also
be developed in the base module.

Best regards,
Jing

On Fri, Oct 21, 2022 at 1:26 PM Jark Wu  wrote:

> How about creating a new repository flink-connector-aws and merging
> dynamodb, kinesis firehouse into it?
> This can reduce the maintenance for complex dependencies and make the
> release easy.
> I think the maintainers of aws-releated connectors are the same people.
>
> Best,
> Jark
>
> > 2022年10月21日 17:41，Chesnay Schepler  写道：
> >
> > I would not go with 2); I think it'd just be messy .
> >
> > Here's another option:
> >
> > Create another repository (aws-connector-base) (following the
> externalization model), add it as a sub-module to the downstream
> repositories, and make it part of the release process of said connector.
> >
> > I.e., we never create a release for aws-connector-bose, but release it
> as part of the connector.
> > This main benefit here is that we'd always be able to make changes to
> the aws-base code without delaying connector releases.
> > I would assume that any added overhead due to _technically_ releasing
> the aws code multiple times to be negligible.
> >
> >
> > On 20/10/2022 22:38, Danny Cranmer wrote:
> >> Hello all,
> >>
> >> Currently we have 2 AWS Flink connectors in the main Flink codebase
> >> (Kinesis Data Streams and Kinesis Data Firehose) and one new
> externalized
> >> connector in progress (DynamoDB). Currently all three of these use
> common
> >> AWS utilities from the flink-connector-aws-base module. Common code
> >> includes client builders, property keys, validation, utils etc.
> >>
> >> Once we externalize the connectors, leaving flink-connector-aws-base in
> the
> >> main Flink repository will restrict our ability to evolve the connectors
> >> quickly. For example, as part of the DynamoDB connector build we are
> >> considering adding a general retry strategy config that can be
> leveraged by
> >> all connectors. We would need to block on Flink 1.17 for this.
> >>
> >> In the past we have tried to keep the AWS SDK version consistent across
> >> connectors, with the externalization this is more likely to diverge.
> >>
> >> Option 1: I propose we create a new repository, flink-connector-aws,
> which
> >> we can move the flink-connector-aws-base module to and create a new
> >> flink-connector-aws-parent to manage SDK versions. Each of the
> externalized
> >> AWS connectors will depend on this new module and parent. Downside is an
> >> additional module to release per Flink version, however I will
> volunteer to
> >> manage this.
> >>
> >> Option 2: We can move the flink-connector-aws-base module and create
> >> flink-connector-parent within the flink-connector-shared-utils repo [2]
> >>
> >> Option 3: We do nothing.
> >>
> >> For option 1+2 we will follow the general externalized connector
> versioning
> >> strategy and rules.
> >>
> >> I am inclined towards option 1, and appreciate feedback from the
> community.
> >>
> >> [1]
> >>
> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
> >> [2] https://github.com/apache/flink-connector-shared-utils
> >>
> >> Thanks,
> >> Danny
> >>
> >
>
>

[jira] [Created] (FLINK-29727) Fall back to flink-conf.yaml if no JOB_MANAGER_RPC_ADDRESS is specified

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-29727:
-

 Summary: Fall back to flink-conf.yaml if no 
JOB_MANAGER_RPC_ADDRESS is specified
 Key: FLINK-29727
 URL: https://issues.apache.org/jira/browse/FLINK-29727
 Project: Flink
  Issue Type: Improvement
  Components: flink-docker
Reporter: Alexander Fedulov
Assignee: Alexander Fedulov


Currently {{docker-enterypoint.sh}} always overrides {{jobmanager.rpc.address}} 
in the {{flink-conf.yaml}} either with the environment variable 
{{JOB_MANAGER_RPC_ADDRESS}} or with a {{hosname}} : 
[link|https://github.com/apache/flink-docker/blob/3c259f46231b97202925a111a8205193c15bbf78/1.15/scala_2.12-java8-ubuntu/docker-entrypoint.sh#L25]
 . This causes, for instance, jobmanager address configured in 
`FlinkContainers` that are based on an existing image (contains 
`docker-entrypoint.sh` in contrast to an image that is built on-the flight from 
flink-dist) to be overridden by the hostname. TMs then fail for connect to the 
JM. A workaround is to use {{TestContainersSettings}} to set 
{{JOB_MANAGER_RPC_ADDRESS }}, which is a suboptimal from the user perspective.

Configuration in flink-conf.yaml should instead be kept if 
JOB_MANAGER_RPC_ADDRESS is not passed explicitly and only overridden by the 
{{hostname}} if nothing was specified in the confit. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Planning Flink 1.17

[Yu Li]

+1 to feature freeze the 1.17 release on 1.17 (smile), and thanks all for
volunteering as our release managers.

Looking forward to the soon-released 1.16.0 and the detailed feature plan
for 1.17.

Best Regards,
Yu


On Fri, 21 Oct 2022 at 20:17, Jing Ge  wrote:

> +1
> The plan looks rational. Thanks for your effort!
>
> Best regards,
> Jing
>
> On Fri, Oct 21, 2022 at 1:16 PM Congxian Qiu 
> wrote:
>
> > Thanks for kicking off the 1.17 release, and volunteers as release
> > managers.
> >
> > +1 to the feature freeze date
> >
> > Best,
> > Congxian
> >
> >
> > Yun Tang  于2022年10月21日周五 17:24写道：
> >
> > > Thanks, Qingsheng and Leonard to kick off the release plan of
> flink-1.17,
> > > which targets a feature freeze date of January 17th 🙂.
> > >
> > > +1 to include Martijn and Matthias as the release managers.
> > >
> > > Best,
> > > Yun Tang
> > > 
> > > From: Piotr Nowojski 
> > > Sent: Friday, October 21, 2022 16:22
> > > To: dev@flink.apache.org 
> > > Subject: Re: [DISCUSS] Planning Flink 1.17
> > >
> > > Hi,
> > >
> > > Thanks for kicking this discussion off Qingsheng and Leonard. I like
> the
> > > January 17th, 2023 touch :)
> > >
> > > 1 for including Matthias Pohl and Martijn Visser as release managers.
> > >
> > > Best,
> > > Piotrek
> > > Always alright
> > >
> > > pt., 21 paź 2022 o 05:55 Yuan Mei  napisał(a):
> > >
> > > > Thanks, Qingsheng for the kicking-off efforts.
> > > >
> > > > 1. January 17th, 2023 as feature freeze data sounds reasonable to me.
> > > > 2. We will input our plan to the wiki link.
> > > >
> > > > Thanks
> > > >
> > > > Best
> > > > Yuan
> > > > Ververica (Alibaba)
> > > >
> > > >
> > > > On Fri, Oct 21, 2022 at 10:38 AM Xingbo Huang 
> wrote:
> > > >
> > > > > Thanks Qingsheng, Leonard and Martijn for starting the discussion
> and
> > > > > volunteering.
> > > > > The timeline proposal sounds reasonable :+1:
> > > > >
> > > > > Best,
> > > > > Xingbo
> > > > >
> > > > > Jark Wu  于2022年10月21日周五 00:17写道：
> > > > >
> > > > > > Thanks for kicking off the 1.17 release.
> > > > > >
> > > > > > Targeting feature freeze on 1/17 for 1.17 release sounds pretty
> > good
> > > to
> > > > > me.
> > > > > > +1 for the volunteers as release managers.
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > > Ververica (Alibaba)
> > > > > >
> > > > > > On Thu, 20 Oct 2022 at 18:09, Matthias Pohl <
> > matthias.p...@aiven.io
> > > > > > .invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for starting the discussion about Flink 1.17. I would be
> > > > > > interested
> > > > > > > in helping out around the release as well.
> > > > > > >
> > > > > > > Best,
> > > > > > > Matthias
> > > > > > >
> > > > > > > On Thu, Oct 20, 2022 at 12:07 PM Xintong Song <
> > > tonysong...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for kicking this off.
> > > > > > > >
> > > > > > > > +1 for the proposed timeline.
> > > > > > > >
> > > > > > > > Also +1 for Qingsheng, Leonard and Martijn as the release
> > > managers.
> > > > > > > Thanks
> > > > > > > > for volunteering.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Xintong
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Oct 20, 2022 at 3:59 PM Martijn Visser <
> > > > > > martijnvis...@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Qingsheng,
> > > > > > > > >
> > > > > > > > > I'm definitely interested in participating as a release
> > manager
> > > > > > again.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > > Martijn
> > > > > > > > >
> > > > > > > > > On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren <
> > > re...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > As we are approaching the official release of Flink 1.16,
> > > it’s
> > > > a
> > > > > > good
> > > > > > > > > time
> > > > > > > > > > to kick off some discussions and march toward 1.17.
> > > > > > > > > >
> > > > > > > > > > - Release managers
> > > > > > > > > >
> > > > > > > > > > Leonard Xu and I would like to volunteer as release
> > managers
> > > > for
> > > > > > > 1.17,
> > > > > > > > > and
> > > > > > > > > > it would be great to have someone else working together
> on
> > > this
> > > > > > > > release.
> > > > > > > > > > Please let us know if you have any interest!
> > > > > > > > > >
> > > > > > > > > > - Timeline
> > > > > > > > > >
> > > > > > > > > > Having 1.16 will be released in the next few days and
> the 4
> > > > > months
> > > > > > > > > release
> > > > > > > > > > cycle after that, we propose to set the feature freezing
> > date
> > > > on
> > > > > > > > *January
> > > > > > > > > > 17th, 2023* (aligned with our version number 1.17 :-)),
> so
> > > that
> > > > > > > > everyone
> > > > > > > > > > could enjoy the holiday season and Chinese new year.
> > > > > > > > > >
> > > > > > > > > > - Wh

Re: [DISCUSS] Planning Flink 1.17

[Jing Ge]

+1
The plan looks rational. Thanks for your effort!

Best regards,
Jing

On Fri, Oct 21, 2022 at 1:16 PM Congxian Qiu  wrote:

> Thanks for kicking off the 1.17 release, and volunteers as release
> managers.
>
> +1 to the feature freeze date
>
> Best,
> Congxian
>
>
> Yun Tang  于2022年10月21日周五 17:24写道：
>
> > Thanks, Qingsheng and Leonard to kick off the release plan of flink-1.17,
> > which targets a feature freeze date of January 17th 🙂.
> >
> > +1 to include Martijn and Matthias as the release managers.
> >
> > Best,
> > Yun Tang
> > 
> > From: Piotr Nowojski 
> > Sent: Friday, October 21, 2022 16:22
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] Planning Flink 1.17
> >
> > Hi,
> >
> > Thanks for kicking this discussion off Qingsheng and Leonard. I like the
> > January 17th, 2023 touch :)
> >
> > 1 for including Matthias Pohl and Martijn Visser as release managers.
> >
> > Best,
> > Piotrek
> > Always alright
> >
> > pt., 21 paź 2022 o 05:55 Yuan Mei  napisał(a):
> >
> > > Thanks, Qingsheng for the kicking-off efforts.
> > >
> > > 1. January 17th, 2023 as feature freeze data sounds reasonable to me.
> > > 2. We will input our plan to the wiki link.
> > >
> > > Thanks
> > >
> > > Best
> > > Yuan
> > > Ververica (Alibaba)
> > >
> > >
> > > On Fri, Oct 21, 2022 at 10:38 AM Xingbo Huang  wrote:
> > >
> > > > Thanks Qingsheng, Leonard and Martijn for starting the discussion and
> > > > volunteering.
> > > > The timeline proposal sounds reasonable :+1:
> > > >
> > > > Best,
> > > > Xingbo
> > > >
> > > > Jark Wu  于2022年10月21日周五 00:17写道：
> > > >
> > > > > Thanks for kicking off the 1.17 release.
> > > > >
> > > > > Targeting feature freeze on 1/17 for 1.17 release sounds pretty
> good
> > to
> > > > me.
> > > > > +1 for the volunteers as release managers.
> > > > >
> > > > > Best,
> > > > > Jark
> > > > > Ververica (Alibaba)
> > > > >
> > > > > On Thu, 20 Oct 2022 at 18:09, Matthias Pohl <
> matthias.p...@aiven.io
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > Thanks for starting the discussion about Flink 1.17. I would be
> > > > > interested
> > > > > > in helping out around the release as well.
> > > > > >
> > > > > > Best,
> > > > > > Matthias
> > > > > >
> > > > > > On Thu, Oct 20, 2022 at 12:07 PM Xintong Song <
> > tonysong...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for kicking this off.
> > > > > > >
> > > > > > > +1 for the proposed timeline.
> > > > > > >
> > > > > > > Also +1 for Qingsheng, Leonard and Martijn as the release
> > managers.
> > > > > > Thanks
> > > > > > > for volunteering.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Xintong
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Oct 20, 2022 at 3:59 PM Martijn Visser <
> > > > > martijnvis...@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Qingsheng,
> > > > > > > >
> > > > > > > > I'm definitely interested in participating as a release
> manager
> > > > > again.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Martijn
> > > > > > > >
> > > > > > > > On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren <
> > re...@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > As we are approaching the official release of Flink 1.16,
> > it’s
> > > a
> > > > > good
> > > > > > > > time
> > > > > > > > > to kick off some discussions and march toward 1.17.
> > > > > > > > >
> > > > > > > > > - Release managers
> > > > > > > > >
> > > > > > > > > Leonard Xu and I would like to volunteer as release
> managers
> > > for
> > > > > > 1.17,
> > > > > > > > and
> > > > > > > > > it would be great to have someone else working together on
> > this
> > > > > > > release.
> > > > > > > > > Please let us know if you have any interest!
> > > > > > > > >
> > > > > > > > > - Timeline
> > > > > > > > >
> > > > > > > > > Having 1.16 will be released in the next few days and the 4
> > > > months
> > > > > > > > release
> > > > > > > > > cycle after that, we propose to set the feature freezing
> date
> > > on
> > > > > > > *January
> > > > > > > > > 17th, 2023* (aligned with our version number 1.17 :-)), so
> > that
> > > > > > > everyone
> > > > > > > > > could enjoy the holiday season and Chinese new year.
> > > > > > > > >
> > > > > > > > > - What we’ll be focusing
> > > > > > > > >
> > > > > > > > > Similar to our previous releases, we’d like to keep an eye
> on
> > > the
> > > > > > > > > timeline, CI stability, release testing, and any
> > communication
> > > > and
> > > > > > > > > coordination across teams and developers. One thing we’d
> like
> > > to
> > > > > > > mention
> > > > > > > > in
> > > > > > > > > particular is compatibility, which is a frequent complaint
> > from
> > > > our
> > > > > > > > > ecosystem developers and users. We encourage all committers
> > to
> > > do
> > > > > an
> > > > > > > > extra
> > > > > > > > > manual

[jira] [Created] (FLINK-29726) Supports hive stddev_samp function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29726:
-

 Summary: Supports hive stddev_samp function by native 
implementation
 Key: FLINK-29726
 URL: https://issues.apache.org/jira/browse/FLINK-29726
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29725) Supports hive std function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29725:
-

 Summary: Supports hive std function by native implementation
 Key: FLINK-29725
 URL: https://issues.apache.org/jira/browse/FLINK-29725
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29724) Supports hive last_value function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29724:
-

 Summary: Supports hive last_value function by native implementation
 Key: FLINK-29724
 URL: https://issues.apache.org/jira/browse/FLINK-29724
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29723) Supports hive first_value function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29723:
-

 Summary: Supports hive first_value function by native 
implementation
 Key: FLINK-29723
 URL: https://issues.apache.org/jira/browse/FLINK-29723
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29722) Supports hive max function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29722:
-

 Summary: Supports hive max function by native implementation
 Key: FLINK-29722
 URL: https://issues.apache.org/jira/browse/FLINK-29722
 Project: Flink
  Issue Type: Improvement
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29721) Supports hive min function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29721:
-

 Summary: Supports hive min function by native implementation
 Key: FLINK-29721
 URL: https://issues.apache.org/jira/browse/FLINK-29721
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29720) Supports hive average function by native implemetatoin

[dalongliu (Jira)]

dalongliu created FLINK-29720:
-

 Summary: Supports hive average function by native implemetatoin
 Key: FLINK-29720
 URL: https://issues.apache.org/jira/browse/FLINK-29720
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29718) Supports hive sum function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29718:
-

 Summary: Supports hive sum function by native implementation
 Key: FLINK-29718
 URL: https://issues.apache.org/jira/browse/FLINK-29718
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29719) Supports hive count function by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29719:
-

 Summary: Supports hive count function by native implementation
 Key: FLINK-29719
 URL: https://issues.apache.org/jira/browse/FLINK-29719
 Project: Flink
  Issue Type: Sub-task
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Create a dedicated aws-base connector repository

[Jark Wu]

How about creating a new repository flink-connector-aws and merging dynamodb, 
kinesis firehouse into it? 
This can reduce the maintenance for complex dependencies and make the release 
easy. 
I think the maintainers of aws-releated connectors are the same people.

Best,
Jark

> 2022年10月21日 17:41，Chesnay Schepler  写道：
> 
> I would not go with 2); I think it'd just be messy .
> 
> Here's another option:
> 
> Create another repository (aws-connector-base) (following the externalization 
> model), add it as a sub-module to the downstream repositories, and make it 
> part of the release process of said connector.
> 
> I.e., we never create a release for aws-connector-bose, but release it as 
> part of the connector.
> This main benefit here is that we'd always be able to make changes to the 
> aws-base code without delaying connector releases.
> I would assume that any added overhead due to _technically_ releasing the aws 
> code multiple times to be negligible.
> 
> 
> On 20/10/2022 22:38, Danny Cranmer wrote:
>> Hello all,
>> 
>> Currently we have 2 AWS Flink connectors in the main Flink codebase
>> (Kinesis Data Streams and Kinesis Data Firehose) and one new externalized
>> connector in progress (DynamoDB). Currently all three of these use common
>> AWS utilities from the flink-connector-aws-base module. Common code
>> includes client builders, property keys, validation, utils etc.
>> 
>> Once we externalize the connectors, leaving flink-connector-aws-base in the
>> main Flink repository will restrict our ability to evolve the connectors
>> quickly. For example, as part of the DynamoDB connector build we are
>> considering adding a general retry strategy config that can be leveraged by
>> all connectors. We would need to block on Flink 1.17 for this.
>> 
>> In the past we have tried to keep the AWS SDK version consistent across
>> connectors, with the externalization this is more likely to diverge.
>> 
>> Option 1: I propose we create a new repository, flink-connector-aws, which
>> we can move the flink-connector-aws-base module to and create a new
>> flink-connector-aws-parent to manage SDK versions. Each of the externalized
>> AWS connectors will depend on this new module and parent. Downside is an
>> additional module to release per Flink version, however I will volunteer to
>> manage this.
>> 
>> Option 2: We can move the flink-connector-aws-base module and create
>> flink-connector-parent within the flink-connector-shared-utils repo [2]
>> 
>> Option 3: We do nothing.
>> 
>> For option 1+2 we will follow the general externalized connector versioning
>> strategy and rules.
>> 
>> I am inclined towards option 1, and appreciate feedback from the community.
>> 
>> [1]
>> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
>> [2] https://github.com/apache/flink-connector-shared-utils
>> 
>> Thanks,
>> Danny
>> 
> 

Re: [DISCUSS] Planning Flink 1.17

[Congxian Qiu]

Thanks for kicking off the 1.17 release, and volunteers as release managers.

+1 to the feature freeze date

Best,
Congxian


Yun Tang  于2022年10月21日周五 17:24写道：

> Thanks, Qingsheng and Leonard to kick off the release plan of flink-1.17,
> which targets a feature freeze date of January 17th 🙂.
>
> +1 to include Martijn and Matthias as the release managers.
>
> Best,
> Yun Tang
> 
> From: Piotr Nowojski 
> Sent: Friday, October 21, 2022 16:22
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] Planning Flink 1.17
>
> Hi,
>
> Thanks for kicking this discussion off Qingsheng and Leonard. I like the
> January 17th, 2023 touch :)
>
> 1 for including Matthias Pohl and Martijn Visser as release managers.
>
> Best,
> Piotrek
> Always alright
>
> pt., 21 paź 2022 o 05:55 Yuan Mei  napisał(a):
>
> > Thanks, Qingsheng for the kicking-off efforts.
> >
> > 1. January 17th, 2023 as feature freeze data sounds reasonable to me.
> > 2. We will input our plan to the wiki link.
> >
> > Thanks
> >
> > Best
> > Yuan
> > Ververica (Alibaba)
> >
> >
> > On Fri, Oct 21, 2022 at 10:38 AM Xingbo Huang  wrote:
> >
> > > Thanks Qingsheng, Leonard and Martijn for starting the discussion and
> > > volunteering.
> > > The timeline proposal sounds reasonable :+1:
> > >
> > > Best,
> > > Xingbo
> > >
> > > Jark Wu  于2022年10月21日周五 00:17写道：
> > >
> > > > Thanks for kicking off the 1.17 release.
> > > >
> > > > Targeting feature freeze on 1/17 for 1.17 release sounds pretty good
> to
> > > me.
> > > > +1 for the volunteers as release managers.
> > > >
> > > > Best,
> > > > Jark
> > > > Ververica (Alibaba)
> > > >
> > > > On Thu, 20 Oct 2022 at 18:09, Matthias Pohl  > > > .invalid>
> > > > wrote:
> > > >
> > > > > Thanks for starting the discussion about Flink 1.17. I would be
> > > > interested
> > > > > in helping out around the release as well.
> > > > >
> > > > > Best,
> > > > > Matthias
> > > > >
> > > > > On Thu, Oct 20, 2022 at 12:07 PM Xintong Song <
> tonysong...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for kicking this off.
> > > > > >
> > > > > > +1 for the proposed timeline.
> > > > > >
> > > > > > Also +1 for Qingsheng, Leonard and Martijn as the release
> managers.
> > > > > Thanks
> > > > > > for volunteering.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 20, 2022 at 3:59 PM Martijn Visser <
> > > > martijnvis...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Qingsheng,
> > > > > > >
> > > > > > > I'm definitely interested in participating as a release manager
> > > > again.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Martijn
> > > > > > >
> > > > > > > On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren <
> re...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > As we are approaching the official release of Flink 1.16,
> it’s
> > a
> > > > good
> > > > > > > time
> > > > > > > > to kick off some discussions and march toward 1.17.
> > > > > > > >
> > > > > > > > - Release managers
> > > > > > > >
> > > > > > > > Leonard Xu and I would like to volunteer as release managers
> > for
> > > > > 1.17,
> > > > > > > and
> > > > > > > > it would be great to have someone else working together on
> this
> > > > > > release.
> > > > > > > > Please let us know if you have any interest!
> > > > > > > >
> > > > > > > > - Timeline
> > > > > > > >
> > > > > > > > Having 1.16 will be released in the next few days and the 4
> > > months
> > > > > > > release
> > > > > > > > cycle after that, we propose to set the feature freezing date
> > on
> > > > > > *January
> > > > > > > > 17th, 2023* (aligned with our version number 1.17 :-)), so
> that
> > > > > > everyone
> > > > > > > > could enjoy the holiday season and Chinese new year.
> > > > > > > >
> > > > > > > > - What we’ll be focusing
> > > > > > > >
> > > > > > > > Similar to our previous releases, we’d like to keep an eye on
> > the
> > > > > > > > timeline, CI stability, release testing, and any
> communication
> > > and
> > > > > > > > coordination across teams and developers. One thing we’d like
> > to
> > > > > > mention
> > > > > > > in
> > > > > > > > particular is compatibility, which is a frequent complaint
> from
> > > our
> > > > > > > > ecosystem developers and users. We encourage all committers
> to
> > do
> > > > an
> > > > > > > extra
> > > > > > > > manual check to see if any public interface is touched before
> > > > > merging a
> > > > > > > PR.
> > > > > > > > We could discuss details in another thread later and update
> the
> > > > > > > > contributing guidelines to list which should be treated as
> > public
> > > > > APIs.
> > > > > > > > Please feel free to raise any discussions if you have
> anything
> > > else
> > > > > to
> > > > > > > > emphasize specifically.
> > > > > > > >
> > > > > > > > - Collecting features
> > > > > > > >
> > > > > > > >

Re: [VOTE] Release 1.16.0, release candidate #2

[Xintong Song]

+1 (binding) for this release candidate

- verified checksum & signature
- build from source
- tried example jobs with standalone cluster, web ui & logs look good
- the new completed jobs & history server features work as expected
- the new hybrid shuffle mode works as expected

Best,

Xintong



On Fri, Oct 21, 2022 at 11:09 AM godfrey he  wrote:

> +1 to make 1.16.0 released as soon as possible,
> it's been more than two months since feature freeze,
> 1.17 already starts kicking off. We can fix the critical bugs in 1.16.1.
>
> Best,
> Godfrey
>
> Xintong Song  于2022年10月21日周五 09:57写道：
> >
> > BTW, missing 1.16.0 is probably not that bad. From my experience, the
> x.y.0
> > releases are usually considered unstable and are mainly for trying out
> > purposes. Most users do not upgrade to the new version in production
> until
> > the x.y.1/2 releases, which are considered more stable. As this bug-fix
> is
> > making 1.16.1 anyway, it barely misses anything.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Fri, Oct 21, 2022 at 5:10 AM Danny Cranmer 
> > wrote:
> >
> > > We had a similar situation for Flink 1.15.1 where a non-regression
> > > "critical" bug was impacting a connector [1]. We decided to not block
> the
> > > release to address this issue. Based on this, I am inclined to agree
> with
> > > Martijn and move forward with the release. This bug is not marked as a
> > > "blocker" and we should respect that.
> > >
> > > I am happy to manage a 1.15.3 or 1.16.1 release follow up to address
> these
> > > concerns.
> > >
> > > [1] https://lists.apache.org/thread/8hxz5n6x2v4c6v8z33hdz51h1bhn768d
> > >
> > > Thanks,
> > > Danny
> > >
> > > On Thu, Oct 20, 2022 at 5:01 PM Martijn Visser <
> martijnvis...@apache.org>
> > > wrote:
> > >
> > > > Taking in this fix would require us to cancel RC2 and create another
> > > > release candidate. We are already long-overdue on the Flink 1.16
> release.
> > > > Given that 1.15.3 is not yet released, it can't be a regression
> compared
> > > to
> > > > the current situation of 1.15.2. The Flink Delta connector is not
> part of
> > > > the ASF Flink community so it can't be considered a blocker from a
> ASF
> > > > Flink community perspective.
> > > >
> > > > I do understand the pain. I'm curious what others think if this is
> worthy
> > > > of cancelling the release candidate.
> > > >
> > > > Thanks, Martijn
> > > >
> > > > On Thu, Oct 20, 2022 at 4:54 PM Krzysztof Chmielewski <
> > > > krzysiek.chmielew...@gmail.com> wrote:
> > > >
> > > > > Thank you all for response,
> > > > > however i think you may miss a bigger context regarding those 3
> > > tickets.
> > > > >
> > > > > Those 3 tickets [29509, 29512, 29627] are part of a bigger thing.
> They
> > > > are
> > > > > fixing 1.15 Sink V2 issue, where Task manager will not start after
> > > > recovery
> > > > > for Sink topology with Global Committer. The problem was described
> by
> > > me
> > > > in
> > > > > this thread [1]. We need all three to fix the problem.
> > > > >
> > > > > All three tickets were merged into 1.15 release branch and will be
> > > > included
> > > > > in 1.15.3 probably, however 1.16 will be missing one fix (29627).
> > > > > In other words, there will be a regression between 1.16 and 1.15.3.
> > > > >
> > > > > Additionally for now this issue is blocking Flink migration for
> Delta
> > > > > connector [2].
> > > > > We need to migrate because Flink 1.14 has another Sink problem with
> > > data
> > > > > loss during Sink Recovery with Global Committer [3] and this one
> most
> > > > > likely will not be fixed since 1.14 support is ending.
> > > > >
> > > > > Forgive my if I'm wrong but what do you mean by " we won't block
> 1.16.0
> > > > on
> > > > > this." Fix is merged so couldn't we just cherry pick 1.16 merge
> commit
> > > to
> > > > > 1.16.0's RC2?
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/otscy199g1l9t3llvo8s2slntyn2r1jc
> > > > > [2] https://github.com/delta-io/connectors/tree/master/flink
> > > > > [3] https://issues.apache.org/jira/browse/FLINK-29589
> > > > >
> > > > > Regards,
> > > > > Krzysztof Chmielewski
> > > > >
> > > > > czw., 20 paź 2022 o 16:13 Xingbo Huang 
> napisał(a):
> > > > >
> > > > > > Hi Krzysztof,
> > > > > >
> > > > > > When I was building rc2, I tried to search whether issues with
> `fix
> > > > > > version` of 1.16.0 have not been closed.
> > > > > > https://issues.apache.org/jira/browse/FLINK-29627 was missed
> because
> > > > the
> > > > > > `fix version` was not marked. I agree with Martijn and Xintong
> that
> > > we
> > > > > > won't block 1.16.0 on this.
> > > > > >
> > > > > > Best,
> > > > > > Xingbo
> > > > > >
> > > > > > Xintong Song  于2022年10月20日周四 18:23写道：
> > > > > >
> > > > > > > Hi Krzysztof,
> > > > > > >
> > > > > > > FLINK-29627 is merged after rc2 being created, that's why it
> > > doesn't
> > > > > > appear
> > > > > > > in the change list. See the commit history of rc2 [1].
> > > > > > >
> > > > > > > It's unfortunate this 

[jira] [Created] (FLINK-29717) Supports hive udaf such as sum/count by native implementation

[dalongliu (Jira)]

dalongliu created FLINK-29717:
-

 Summary: Supports hive udaf  such as sum/count by native 
implementation 
 Key: FLINK-29717
 URL: https://issues.apache.org/jira/browse/FLINK-29717
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive, Table SQL / Runtime
Reporter: dalongliu
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Cassandra] source connector

[Etienne Chauchot]

Hi,

Yes sure, if there is a consensus on moving, just tell me where to move 
my PR to.


Best

Etienne

Le 19/10/2022 à 18:20, Alexander Fedulov a écrit :

Hi Etienne,

thanks for your contribution. In light of the current efforts to
externalize connectors, do you think we could maybe combine the new
implementation with moving it into an external repository instead of
merging into Flink main?

Best,
Alexander Fedulov

On Fri, Oct 14, 2022 at 4:18 PM Etienne Chauchot 
wrote:


Hi all,

As promised, I have developed the Cassandra source connector based on
FLIP-27. I've just submitted the
PR:https://github.com/apache/flink/pull/21073


Best

Etienne

Re: [DISCUSS] Create a dedicated aws-base connector repository

[Chesnay Schepler]

I would not go with 2); I think it'd just be messy .

Here's another option:

Create another repository (aws-connector-base) (following the 
externalization model), add it as a sub-module to the downstream 
repositories, and make it part of the release process of said connector.


I.e., we never create a release for aws-connector-bose, but release it 
as part of the connector.
This main benefit here is that we'd always be able to make changes to 
the aws-base code without delaying connector releases.
I would assume that any added overhead due to _technically_ releasing 
the aws code multiple times to be negligible.



On 20/10/2022 22:38, Danny Cranmer wrote:

Hello all,

Currently we have 2 AWS Flink connectors in the main Flink codebase
(Kinesis Data Streams and Kinesis Data Firehose) and one new externalized
connector in progress (DynamoDB). Currently all three of these use common
AWS utilities from the flink-connector-aws-base module. Common code
includes client builders, property keys, validation, utils etc.

Once we externalize the connectors, leaving flink-connector-aws-base in the
main Flink repository will restrict our ability to evolve the connectors
quickly. For example, as part of the DynamoDB connector build we are
considering adding a general retry strategy config that can be leveraged by
all connectors. We would need to block on Flink 1.17 for this.

In the past we have tried to keep the AWS SDK version consistent across
connectors, with the externalization this is more likely to diverge.

Option 1: I propose we create a new repository, flink-connector-aws, which
we can move the flink-connector-aws-base module to and create a new
flink-connector-aws-parent to manage SDK versions. Each of the externalized
AWS connectors will depend on this new module and parent. Downside is an
additional module to release per Flink version, however I will volunteer to
manage this.

Option 2: We can move the flink-connector-aws-base module and create
flink-connector-parent within the flink-connector-shared-utils repo [2]

Option 3: We do nothing.

For option 1+2 we will follow the general externalized connector versioning
strategy and rules.

I am inclined towards option 1, and appreciate feedback from the community.

[1]
https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
[2] https://github.com/apache/flink-connector-shared-utils

Thanks,
Danny

Re: [DISCUSS] Planning Flink 1.17

[Yun Tang]

Thanks, Qingsheng and Leonard to kick off the release plan of flink-1.17, which 
targets a feature freeze date of January 17th 🙂.

+1 to include Martijn and Matthias as the release managers.

Best,
Yun Tang

From: Piotr Nowojski 
Sent: Friday, October 21, 2022 16:22
To: dev@flink.apache.org 
Subject: Re: [DISCUSS] Planning Flink 1.17

Hi,

Thanks for kicking this discussion off Qingsheng and Leonard. I like the
January 17th, 2023 touch :)

1 for including Matthias Pohl and Martijn Visser as release managers.

Best,
Piotrek
Always alright

pt., 21 paź 2022 o 05:55 Yuan Mei  napisał(a):

> Thanks, Qingsheng for the kicking-off efforts.
>
> 1. January 17th, 2023 as feature freeze data sounds reasonable to me.
> 2. We will input our plan to the wiki link.
>
> Thanks
>
> Best
> Yuan
> Ververica (Alibaba)
>
>
> On Fri, Oct 21, 2022 at 10:38 AM Xingbo Huang  wrote:
>
> > Thanks Qingsheng, Leonard and Martijn for starting the discussion and
> > volunteering.
> > The timeline proposal sounds reasonable :+1:
> >
> > Best,
> > Xingbo
> >
> > Jark Wu  于2022年10月21日周五 00:17写道：
> >
> > > Thanks for kicking off the 1.17 release.
> > >
> > > Targeting feature freeze on 1/17 for 1.17 release sounds pretty good to
> > me.
> > > +1 for the volunteers as release managers.
> > >
> > > Best,
> > > Jark
> > > Ververica (Alibaba)
> > >
> > > On Thu, 20 Oct 2022 at 18:09, Matthias Pohl  > > .invalid>
> > > wrote:
> > >
> > > > Thanks for starting the discussion about Flink 1.17. I would be
> > > interested
> > > > in helping out around the release as well.
> > > >
> > > > Best,
> > > > Matthias
> > > >
> > > > On Thu, Oct 20, 2022 at 12:07 PM Xintong Song  >
> > > > wrote:
> > > >
> > > > > Thanks for kicking this off.
> > > > >
> > > > > +1 for the proposed timeline.
> > > > >
> > > > > Also +1 for Qingsheng, Leonard and Martijn as the release managers.
> > > > Thanks
> > > > > for volunteering.
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Oct 20, 2022 at 3:59 PM Martijn Visser <
> > > martijnvis...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Qingsheng,
> > > > > >
> > > > > > I'm definitely interested in participating as a release manager
> > > again.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren 
> > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > As we are approaching the official release of Flink 1.16, it’s
> a
> > > good
> > > > > > time
> > > > > > > to kick off some discussions and march toward 1.17.
> > > > > > >
> > > > > > > - Release managers
> > > > > > >
> > > > > > > Leonard Xu and I would like to volunteer as release managers
> for
> > > > 1.17,
> > > > > > and
> > > > > > > it would be great to have someone else working together on this
> > > > > release.
> > > > > > > Please let us know if you have any interest!
> > > > > > >
> > > > > > > - Timeline
> > > > > > >
> > > > > > > Having 1.16 will be released in the next few days and the 4
> > months
> > > > > > release
> > > > > > > cycle after that, we propose to set the feature freezing date
> on
> > > > > *January
> > > > > > > 17th, 2023* (aligned with our version number 1.17 :-)), so that
> > > > > everyone
> > > > > > > could enjoy the holiday season and Chinese new year.
> > > > > > >
> > > > > > > - What we’ll be focusing
> > > > > > >
> > > > > > > Similar to our previous releases, we’d like to keep an eye on
> the
> > > > > > > timeline, CI stability, release testing, and any communication
> > and
> > > > > > > coordination across teams and developers. One thing we’d like
> to
> > > > > mention
> > > > > > in
> > > > > > > particular is compatibility, which is a frequent complaint from
> > our
> > > > > > > ecosystem developers and users. We encourage all committers to
> do
> > > an
> > > > > > extra
> > > > > > > manual check to see if any public interface is touched before
> > > > merging a
> > > > > > PR.
> > > > > > > We could discuss details in another thread later and update the
> > > > > > > contributing guidelines to list which should be treated as
> public
> > > > APIs.
> > > > > > > Please feel free to raise any discussions if you have anything
> > else
> > > > to
> > > > > > > emphasize specifically.
> > > > > > >
> > > > > > > - Collecting features
> > > > > > >
> > > > > > > We'll create a wiki page under this directory[1] for collecting
> > new
> > > > > > > features targeted in 1.17 as we always did before to give
> > everyone
> > > an
> > > > > > > overview and track the progress. Please don’t hesitate to share
> > > your
> > > > > > ideas
> > > > > > > on the page. In the meantime, we’d like to kindly invite our
> > > > committers
> > > > > > to
> > > > > > > think about and plan what we could deliver to developers and
> > users
> > > in
> > > > > > this
> > > > > > > release.
> > > > > > >
> > > > > > > Looking

dev@flink.apache.org

[Chesnay Schepler]

Of course we can (and will).

That will happen at a later time once we get a bit of use out of it to 
iterate faster.



On 20/10/2022 18:23, Steven Wu wrote:

Chesnay, thanks for the write-up. very helpful!

Regarding the parent pom, I am wondering if it can be published to the
`org.apache.flink` group?


 io.github.zentol.flink
 flink-connector-parent
 1.0


On Mon, Oct 17, 2022 at 5:52 AM Chesnay Schepler  wrote:


https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development

On 17/10/2022 13:13, Chesnay Schepler wrote:

The vote has passed unanimously.

+1 Votes:
- Danny (binding)
- Martijn (binding)
- Ferenc (non-binding)
- Thomas (binding)
- Ryan (non-binding)
- Jing (non-binding)
- Matthias (binding)

I will now document this in the wiki and start working on the release
scripts.

On 12/10/2022 15:12, Chesnay Schepler wrote:

Since the discussion
(https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo)
has stalled a bit but we need a conclusion to move forward I'm
opening a vote.

Proposal summary:

1) Branch model
1.1) The default branch is called "main" and used for the next major
iteration.
1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2)
1.3) Branches are not specific to a Flink version. (i.e., no v3.2-1.15)

2) Versioning
2.1) Source releases: major.minor.patch
2.2) Jar artifacts: major.minor.match-flink-major.flink-minor
(This may imply releasing the exact same connector jar multiple times
under different versions)

3) Flink compatibility
3.1) The Flink versions supported by the project (last 2 major Flink
versions) must be supported.
3.2) How this is achived is left to the connector, as long as it
conforms to the rest of the proposal.

4) Support
4.1) The last 2 major connector releases are supported with only the
latter receiving additional features, with the following exceptions:
4.1.a) If the older major connector version does not support any
currently supported Flink version, then it is no longer supported.
4.1.b) If the last 2 major versions do not cover all supported Flink
versions, then the latest connector version that supports the older
Flink version /additionally /gets patch support.
4.2) For a given major connector version only the latest minor
version is supported.
(This means if 1.1.x is released there will be no more 1.0.x release)


I'd like to clarify that these won't be set in stone for eternity.
We should re-evaluate how well this model works over time and adjust
it accordingly, consistently across all connectors.
I do believe that as is this strikes a good balance between
maintainability for us and clarity to users.


Voting schema:

Consensus, committers have binding votes, open for at least 72 hours.

[jira] [Created] (FLINK-29716) Separate slf4j jar in the lib folder from the distribution

[Alexis Sarda-Espinosa (Jira)]

Alexis Sarda-Espinosa created FLINK-29716:
-

 Summary: Separate slf4j jar in the lib folder from the distribution
 Key: FLINK-29716
 URL: https://issues.apache.org/jira/browse/FLINK-29716
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.15.2
Reporter: Alexis Sarda-Espinosa


Flink's binary distribution includes several jars under the {{lib}} folder, 
which has individual jars for all log4j artifacts. This makes it relatively 
easy to swap out those logging jars when necessary, for example when critical 
vulnerabilities are found (as was recently the case).

With SLF4J 2.+, some breaking changes mean that many implementations are not 
directly backwards compatible, see for example the [notes for 
log4j2|https://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html]. This 
means that, in the future, if swapping logging jars were necessary, the SLF4J 
jar might have to be changed as well.

Right now the SLF4J jar is not included separately in the distribution, I 
believe it's packed inside the {{flink-dist}} jar, although I'm not sure. It 
would be better to separate that as it is done for the default log4j2 jars.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] REST API to suspend & resume checkpointing

[Piotr Nowojski]

Hi Saurabh,

> 1. Stop with native savepoint does solve any races and produces a
> predictable restoration point, but producing a self-contained snapshot and
> using CLAIM mode in re-running is not necessary here and adds latency.

Why do you think it adds latency? CLAIM mode is actually the fastest one
with zero additional overheads. Native savepoints are almost an equivalent
of checkpoints.

Best,
Piotrek


pt., 21 paź 2022 o 00:51 Saurabh Kaul  napisał(a):

> Hi, thanks for the quick responses,
>
> I think a stop-with-checkpoint idea is overlapping well with the
> requirements.
>
> 1. Stop with native savepoint does solve any races and produces a
> predictable restoration point, but producing a self-contained snapshot and
> using CLAIM mode in re-running is not necessary here and adds latency.
> Stop-with-checkpoint doesn't have these issues. It adds some downtime in
> waiting for a checkpoint to be completed but reduces replay time in the new
> cluster which is a good trade-off. Since in this scenario of job migration
> the job and/or job configuration is not changing; it should ideally be as
> fast as a regular failover scenario (like a TM going down).
> 2. Taking complete ownership of triggering checkpoints and making them more
> configurable could be feasible but are less effective comparatively in
> terms of stopping the job for the primary purpose of low-downtime migration
> of the job. Stop-with-checkpoint solves it more directly.
>
> Looking forward to hearing thoughts on this.
>
> On Thu, Oct 20, 2022 at 3:31 AM Piotr Nowojski 
> wrote:
>
> > Hi Saurabh,
> >
> > Thanks for reaching out with the proposal. I have some mixed feelings
> about
> > this for a couple of reasons:
> >
> > 1. It sounds like the core problem that you are describing is the race
> > condition between shutting down the cluster and completion of new
> > checkpoints. My first thought would be as Jing's, why don't you use
> > stop-with-savepoint? Especially the native savepoint? You can recover
> from
> > it using --claim mode, so the whole process should be quite fast
> actually.
> > 2. The same issue, not knowing the latest completed checkpoint id,
> plagued
> > us with some internal tests for quite a bit, so maybe this would also be
> > worth considering to address instead? Like leaving in some text file the
> > last completed checkpoint id? Or providing a way to read this from some
> > existing metadata files? However in our tests we actually fixed/worked
> > around that with manually triggering of checkpoints. The predecessor of
> > FLINK-27101 [1], FLINK-24280 [2], was implemented to address this exact
> > issue.  Which brings me to...
> > 3. You could actually just use the REST API to trigger all checkpoints
> > manually. The idea behind FLINK-27101 [1] was to add full flexibility to
> > the users, without adding much complexity to the system. If we start
> adding
> > more REST calls to control checkpointing behaviour it would complicate
> the
> > system.
> > 4. If at all, I would think more towards a more generic idea of
> dynamically
> > reconfiguring the system. We could provide a generic way to dynamically
> > change configuration options. We wouldn't be able to support all
> > configurations, and furthermore, each "dynamic" option would have to be
> > handled/passed down to and through the system differently, BUT we
> wouldn't
> > have to do all of that at once. We could start with a very limited set of
> > dynamic options, for example just with the checkpointing interval. This
> > must have been considered/discussed before, so I might be missing lots of
> > things.
> > 5. Another direction, if 1. is not an option for some reason, is to
> provide
> > a stop-with-checkpoint feature?
> >
> > Best Piotrek
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-27101
> > [2] https://issues.apache.org/jira/browse/FLINK-24280
> >
> > czw., 20 paź 2022 o 11:53 Jing Ge  napisał(a):
> >
> > > Hi Saurabh,
> > >
> > > In general, it is always good to add new features. I am not really sure
> > if
> > > I understood your requirement. I guess it will be too long for you to
> > > resume the job with a created savepoint in the new stand-by Flink
> > cluster.
> > > But if it would be acceptable to you, you should not have the issue you
> > > mentioned with the checkpoint. Speaking of checkpoint, if the
> checkpoint
> > > interval were set properly, it should be fine even if in some rare
> cases
> > > the last checkpoint was partially completed and is not selected.
> Another
> > > option could be to trigger a manual checkpoint and then use that one to
> > > resume the job to maintain the low downtime.
> > >
> > > Best regards,
> > > JIng
> > >
> > > On Thu, Oct 20, 2022 at 7:53 AM Saurabh Kaul 
> wrote:
> > >
> > > > Hey everyone,
> > > >
> > > > I will create a FLIP, but wanted to gauge community opinion first.
> The
> > > > motivation is that production Flink applications frequently need to
> go
> > > > through node/image patching 

[jira] [Created] (FLINK-29715) Expose max_parallelism in JSON plan

[Gyula Fora (Jira)]

Gyula Fora created FLINK-29715:
--

 Summary: Expose max_parallelism in JSON plan
 Key: FLINK-29715
 URL: https://issues.apache.org/jira/browse/FLINK-29715
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / REST
Reporter: Gyula Fora
Assignee: Gyula Fora


The JobGraph json plan currently only contains vertex parallelism but not the 
max_parallelism. This could be very useful information to also show on the UI 
for debugging data skew/performance issues or for any tooling that relies on 
the jobgraph information gathered from the rest endpoint.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Planning Flink 1.17

[Piotr Nowojski]

Hi,

Thanks for kicking this discussion off Qingsheng and Leonard. I like the
January 17th, 2023 touch :)

1 for including Matthias Pohl and Martijn Visser as release managers.

Best,
Piotrek
Always alright

pt., 21 paź 2022 o 05:55 Yuan Mei  napisał(a):

> Thanks, Qingsheng for the kicking-off efforts.
>
> 1. January 17th, 2023 as feature freeze data sounds reasonable to me.
> 2. We will input our plan to the wiki link.
>
> Thanks
>
> Best
> Yuan
> Ververica (Alibaba)
>
>
> On Fri, Oct 21, 2022 at 10:38 AM Xingbo Huang  wrote:
>
> > Thanks Qingsheng, Leonard and Martijn for starting the discussion and
> > volunteering.
> > The timeline proposal sounds reasonable :+1:
> >
> > Best,
> > Xingbo
> >
> > Jark Wu  于2022年10月21日周五 00:17写道：
> >
> > > Thanks for kicking off the 1.17 release.
> > >
> > > Targeting feature freeze on 1/17 for 1.17 release sounds pretty good to
> > me.
> > > +1 for the volunteers as release managers.
> > >
> > > Best,
> > > Jark
> > > Ververica (Alibaba)
> > >
> > > On Thu, 20 Oct 2022 at 18:09, Matthias Pohl  > > .invalid>
> > > wrote:
> > >
> > > > Thanks for starting the discussion about Flink 1.17. I would be
> > > interested
> > > > in helping out around the release as well.
> > > >
> > > > Best,
> > > > Matthias
> > > >
> > > > On Thu, Oct 20, 2022 at 12:07 PM Xintong Song  >
> > > > wrote:
> > > >
> > > > > Thanks for kicking this off.
> > > > >
> > > > > +1 for the proposed timeline.
> > > > >
> > > > > Also +1 for Qingsheng, Leonard and Martijn as the release managers.
> > > > Thanks
> > > > > for volunteering.
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Oct 20, 2022 at 3:59 PM Martijn Visser <
> > > martijnvis...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Qingsheng,
> > > > > >
> > > > > > I'm definitely interested in participating as a release manager
> > > again.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren 
> > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > As we are approaching the official release of Flink 1.16, it’s
> a
> > > good
> > > > > > time
> > > > > > > to kick off some discussions and march toward 1.17.
> > > > > > >
> > > > > > > - Release managers
> > > > > > >
> > > > > > > Leonard Xu and I would like to volunteer as release managers
> for
> > > > 1.17,
> > > > > > and
> > > > > > > it would be great to have someone else working together on this
> > > > > release.
> > > > > > > Please let us know if you have any interest!
> > > > > > >
> > > > > > > - Timeline
> > > > > > >
> > > > > > > Having 1.16 will be released in the next few days and the 4
> > months
> > > > > > release
> > > > > > > cycle after that, we propose to set the feature freezing date
> on
> > > > > *January
> > > > > > > 17th, 2023* (aligned with our version number 1.17 :-)), so that
> > > > > everyone
> > > > > > > could enjoy the holiday season and Chinese new year.
> > > > > > >
> > > > > > > - What we’ll be focusing
> > > > > > >
> > > > > > > Similar to our previous releases, we’d like to keep an eye on
> the
> > > > > > > timeline, CI stability, release testing, and any communication
> > and
> > > > > > > coordination across teams and developers. One thing we’d like
> to
> > > > > mention
> > > > > > in
> > > > > > > particular is compatibility, which is a frequent complaint from
> > our
> > > > > > > ecosystem developers and users. We encourage all committers to
> do
> > > an
> > > > > > extra
> > > > > > > manual check to see if any public interface is touched before
> > > > merging a
> > > > > > PR.
> > > > > > > We could discuss details in another thread later and update the
> > > > > > > contributing guidelines to list which should be treated as
> public
> > > > APIs.
> > > > > > > Please feel free to raise any discussions if you have anything
> > else
> > > > to
> > > > > > > emphasize specifically.
> > > > > > >
> > > > > > > - Collecting features
> > > > > > >
> > > > > > > We'll create a wiki page under this directory[1] for collecting
> > new
> > > > > > > features targeted in 1.17 as we always did before to give
> > everyone
> > > an
> > > > > > > overview and track the progress. Please don’t hesitate to share
> > > your
> > > > > > ideas
> > > > > > > on the page. In the meantime, we’d like to kindly invite our
> > > > committers
> > > > > > to
> > > > > > > think about and plan what we could deliver to developers and
> > users
> > > in
> > > > > > this
> > > > > > > release.
> > > > > > >
> > > > > > > Looking forward to working with you all in the coming 1.17
> > release!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Qingsheng Ren and Leonard Xu
> > > > > > > Ververica (Alibaba)
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan
> > > > > >

Re: SQL Gateway and SQL Client

[Shengkai Fang]

Sorry for the late response.

In the next version(Flink 1.17), we plan to support the SQL Client to
submit the statement to the Flink SQL Gateway. The FLINK-29486
 is the first step to
remove the usage of the `Parser` in the client side, which needs to read
the table schema during the converting sql node to operation. I think
the authentication
module is not part of our plan in 1.17 because of the busy work. I think
we'll start the design at the end of the release-1.17.
But could you share more details about the requirements of the
authentication?
- Do you use the kerberos or delegation token or password to do the
authentication?
- After the authentication, do you need the sql gateway to submit the job
on behalf of the client?
- ...

For detailed implementation, I think Hive and Presto are good examples to
dig in.  If you have some thoughts about the authentication module, please
let me know.

Best,
Shengkai

Alexey Leonov-Vendrovskiy  于2022年10月19日周三 00:37写道：

> Thank you for the response, Yuxia!
>
> Shengkai, I would like to learn more about nearest and a bit more distant
> plans about development of the SQL Gateway and the SQL Client.
> Do you have a description of the work planned or maybe can share general
> thoughts about the Authentication module, or Persistent Gateway.
> How can the authentication part be addressed on the SQL Client side?
>
> Regards,
> -Alexey
>
>
> On Wed, Oct 12, 2022 at 11:24 PM yuxia 
> wrote:
>
>> > In what Flink’s release the connection from SQL Client to the Gateway is
>> expected to be added?
>> Flink 1.17
>>
>> > “Authentication module” (2) and “Persistent Gateway” (4) as
>> possible future work. Were there any recent discussions on these subjects?
>> No recent discussions on these subjects, but I think it'll come in Flink
>> 1.17
>>
>> > Another related topic: are there ideas around making SQL Gateway a
>> multi-tenant
>> component?
>> Yes.
>>
>> Shengkaiis the maintainer of SQL Client and SQL gateway, maybe he can
>> provide more information.
>>
>>
>>
>> Best regards,
>> Yuxia
>>
>> - 原始邮件 -
>> 发件人: "Alexey Leonov-Vendrovskiy" 
>> 收件人: "dev" 
>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08
>> 主题: SQL Gateway and SQL Client
>>
>> Hi all,
>>
>> I’m Alexey from Confluent. This is my first email in this discussion list.
>> I’m rather new to Flink, and to local customs of communication. I want to
>> dive deeper and hopefully get more involved over time.
>>
>> Currently I have a few questions around SQL Gateway and SQL Client.
>> Specifically I wanted to learn what is the vision around the nearest
>> future
>> of these two components.
>>
>> In what Flink’s release the connection from SQL Client to the Gateway is
>> expected to be added? I was looking at
>> https://issues.apache.org/jira/browse/FLINK-29486, and recently it got
>> renamed from “Enable SQL Client to Connect SQL Gateway in Remote Mode” to
>> “Introduce Client Parser to get statement type”.  I did some search, but
>> didn’t find a good place where the client's work in this direction is
>> discussed or tracked.
>>
>> A couple questions about the SQL Gateway. The FLIP-91
>> <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework
>> >
>> mentions “Authentication module” (2) and “Persistent Gateway” (4) as
>> possible future work. Were there any recent discussions on these subjects?
>> Or maybe there are some ideas how to move these directions forward?
>> Another
>> related topic: are there ideas around making SQL Gateway a multi-tenant
>> component?
>>
>> Thank you,
>>
>> Alexey
>>
>

[jira] [Created] (FLINK-29714) Merge TableWrite and TableCompact into one interface

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29714:
---

 Summary: Merge TableWrite and TableCompact into one interface
 Key: FLINK-29714
 URL: https://issues.apache.org/jira/browse/FLINK-29714
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Caizhi Weng
 Fix For: table-store-0.3.0


To make sure that full compaction is triggered constantly for every written 
bucket regardless of failure, we need to add {{compact}} interface to 
{{TableWrite}} so that Flink sink operators can trigger compaction when needed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Modify savepoints in Flink

[Sriram Ganesh]

Sorry. Ignore. Moved this question to the user.

On Fri, Oct 21, 2022 at 1:22 PM Sriram Ganesh  wrote:

> Hi All,
>
> I am working on a scenario where I need to modify the existing savepoint
> operator state. Ex: Wanted to remove some offset of the savepoint.
>
> What is the better practice for these scenarios?. Could you please help me
> with any example as such?
>
> Thanks in advance.
>
> --
> *Sriram G*
> *Tech*
>
>

-- 
*Sriram G*
*Tech*

[jira] [Created] (FLINK-29713) Kubernetes operator should restart failed jobs

[Peter Vary (Jira)]

Peter Vary created FLINK-29713:
--

 Summary: Kubernetes operator should restart failed jobs
 Key: FLINK-29713
 URL: https://issues.apache.org/jira/browse/FLINK-29713
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Peter Vary


It would be good to have the possibility to restart the Flink Application if it 
goes to {{FAILED}} state.
This could be used to restart, and reconfigure the job dynamically in the 
application {{main}} method if the current application can not handle the 
incoming data



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Modify savepoints in Flink

[Sriram Ganesh]

Hi All,

I am working on a scenario where I need to modify the existing savepoint
operator state. Ex: Wanted to remove some offset of the savepoint.

What is the better practice for these scenarios?. Could you please help me
with any example as such?

Thanks in advance.

-- 
*Sriram G*
*Tech*