[ANNOUNCE] Apache flink-connector-mongodb v1.0.0 released
The Apache Flink community is very happy to announce the release of Apache flink-connector-mongodb v1.0.0 Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352386 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Danny
[ANNOUNCE] Apache flink-connector-aws v4.1.0 released
The Apache Flink community is very happy to announce the release of Apache flink-connector-aws v4.1.0 Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352646 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Danny
[DISCUSS] Status of Statefun Project
Hi everyone, I want to open a discussion on the status of the Statefun Project [1] in Apache Flink. As you might have noticed, there hasn't been much development over the past months in the Statefun repository [2]. There is currently a lack of active contributors and committers who are able to help with the maintenance of the project. In order to improve the situation, we need to solve the lack of committers and the lack of contributors. On the lack of committers: 1. Ideally, there are some of the current Flink committers who have the bandwidth and can help with reviewing PRs and merging them. 2. If that's not an option, it could be a consideration that current committers only approve and review PRs, that are approved by those who are willing to contribute to Statefun and if the CI passes On the lack of contributors: 3. Next to having this discussion on the Dev and User mailing list, we can also create a blog with a call for new contributors on the Flink project website, send out some tweets on the Flink / Statefun twitter accounts, post messages on Slack etc. In that message, we would inform how those that are interested in contributing can start and where they could reach out for more information. There's also option 4. where a group of interested people would split Statefun from the Flink project and make it a separate top level project under the Apache Flink umbrella (similar as recently has happened with Flink Table Store, which has become Apache Paimon). If we see no improvements in the coming period, we should consider sunsetting Statefun and communicate that clearly to the users. I'm looking forward to your thoughts. Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-statefun-docs-master/ [2] https://github.com/apache/flink-statefun
[ANNOUNCE] Starting with Flink 1.18 Release Sync
Hi everyone, As a fresh start of the Flink release 1.18, I'm happy to share with you that the first release sync meeting of 1.18 will happen tomorrow on Tuesday, April 4th at 10am (UTC+2) / 4pm (UTC+8). Welcome and feel free to join us and share your ideas about the new release cycle! Details of joining the release sync can be found in the 1.18 release wiki page [1]. All contributors are invited to update the same wiki page [1] and include features targeting the 1.18 release. Looking forward to seeing you all in the meeting! [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release Best regards, Jing, Konstantin, Sergey and Qingsheng
How to set reblance in Flink Sql like Streaming api?
For example. I create a kafka source to subscribe the topic that have one partition and set the default parallelism of the job to 6.The next operator of kafka source is that lookup join a mysql table.However, the relationship between the kafka Source and the Lookup join operator is Forward, so only one subtask in the Lookup join operator can receive data.I want to set the relationship between the kafka Source and the Lookup join operator is reblance so that all subtask in Lookup join operator can recevie data. Env: Flink version:1.15.1 -- Best， Hjw
Re: [DISCUSS] Status of Statefun Project
Thanks for bringing this up. I'm currently using Statefun, and I've made a few small code contributions over time. All of my PRs have been merged into master and most have been released, but a few haven't been part of a release yet. Most recently, I helped upgrade Statefun to be compatible with Flink 1.15.2, which was merged last October but hasn't been released. (And, of course, there have been more Flink releases since then.) IMO, the main thing driving the need for ongoing Statefun releases -- even in the absence of any new feature development -- is that there is typically a bit of work to do to make Statefun compatible with each new Flink release. This usually involves updating dependency versions and sometimes some simple code changes, a common example being adapting to changes in Flink config parameters that have changed from, say, delimited strings to arrays. I'd be happy to continue to make the necessary changes to Statefun to be compatible with each new Flink release, but I don't have the committer rights that would allow me to release the code. On Mon, Apr 3, 2023 at 5:02 AM Martijn Visser wrote: > Hi everyone, > > I want to open a discussion on the status of the Statefun Project [1] in > Apache Flink. As you might have noticed, there hasn't been much development > over the past months in the Statefun repository [2]. There is currently a > lack of active contributors and committers who are able to help with the > maintenance of the project. > > In order to improve the situation, we need to solve the lack of committers > and the lack of contributors. > > On the lack of committers: > > 1. Ideally, there are some of the current Flink committers who have the > bandwidth and can help with reviewing PRs and merging them. > 2. If that's not an option, it could be a consideration that current > committers only approve and review PRs, that are approved by those who are > willing to contribute to Statefun and if the CI passes > > On the lack of contributors: > > 3. Next to having this discussion on the Dev and User mailing list, we can > also create a blog with a call for new contributors on the Flink project > website, send out some tweets on the Flink / Statefun twitter accounts, > post messages on Slack etc. In that message, we would inform how those that > are interested in contributing can start and where they could reach out for > more information. > > There's also option 4. where a group of interested people would split > Statefun from the Flink project and make it a separate top level project > under the Apache Flink umbrella (similar as recently has happened with > Flink Table Store, which has become Apache Paimon). > > If we see no improvements in the coming period, we should consider > sunsetting Statefun and communicate that clearly to the users. > > I'm looking forward to your thoughts. > > Best regards, > > Martijn > > [1] https://nightlies.apache.org/flink/flink-statefun-docs-master/ > [2] https://github.com/apache/flink-statefun >
Re: How to set reblance in Flink Sql like Streaming api?
Hi, hiw, IMO, I think the parallelism 1 is enough for you job if we do not consider the sink. I do not know why you need set the lookup join operator's parallelism to 6. The SQL planner will help us to decide the type of the edge and we can not change it. Maybe you could share the Execution graph to provide more information. Best, Hang hjw 于2023年4月4日周二 00:37写道： > For example. I create a kafka source to subscribe the topic that have > one partition and set the default parallelism of the job to 6.The next > operator of kafka source is that lookup join a mysql table.However, the > relationship between the kafka Source and the Lookup join operator is > Forward, so only one subtask in the Lookup join operator can receive data.I > want to set the relationship between the kafka Source and the Lookup join > operator is reblance so that all subtask in Lookup join operator can > recevie data. > > Env: > Flink version:1.15.1 > > > -- > Best， > Hjw >
Re: How to set reblance in Flink Sql like Streaming api?
Hi hjw To rescale data for dim join, I think you can use `partition by` in sql before `dim join` which will redistribute data by specific column. In addition, you can add cache for `dim table` to improve performance too. Best, Shammon FY On Tue, Apr 4, 2023 at 10:28 AM Hang Ruan wrote: > Hi, hiw, > > IMO, I think the parallelism 1 is enough for you job if we do not consider > the sink. I do not know why you need set the lookup join operator's > parallelism to 6. > The SQL planner will help us to decide the type of the edge and we can not > change it. > Maybe you could share the Execution graph to provide more information. > > Best, > Hang > > hjw 于2023年4月4日周二 00:37写道： > >> For example. I create a kafka source to subscribe the topic that have >> one partition and set the default parallelism of the job to 6.The next >> operator of kafka source is that lookup join a mysql table.However, the >> relationship between the kafka Source and the Lookup join operator is >> Forward, so only one subtask in the Lookup join operator can receive data.I >> want to set the relationship between the kafka Source and the Lookup join >> operator is reblance so that all subtask in Lookup join operator can >> recevie data. >> >> Env: >> Flink version:1.15.1 >> >> >> -- >> Best， >> Hjw >> >
