I created a pull request for this: [FLINK-31619] Upgrade Stateful Functions to Flink 1.16.1 by galenwarren · Pull Request #331 · apache/flink-statefun (github.com) <https://github.com/apache/flink-statefun/pull/331>.
JIRA is here: [FLINK-31619] Upgrade Stateful Functions to Flink 1.16.1 - ASF JIRA (apache.org) <https://issues.apache.org/jira/browse/FLINK-31619?filter=-1>. Statefun references 1.16.2, despite the title -- that version has come out since the issue was created. I figured out how to run all the playground tests locally, but it took a bit of reworking of the playground setup with respect to Docker; specifically, the Docker contexts used to build the example functions needed to be broadened (i.e. moved up the tree) so that, if needed, local artifacts/code can be accessed from within the containers at build time. Then I made the Docker compose.yml configurable through environment variables to allow for them to run in either the original manner -- i.e. pulling artifacts from public repos -- or in a "local" mode, where artifacts are pulled from local builds. This process is a cleaner if the playground is a subfolder of the flink-statefun project rather than be its own repository (flink-statefun-playground), because then all the relative paths between the playground files and the build artifacts are fixed. So, I'd like to propose to move the playground files, modified as described above, to flink-statefun/playground and retire flink-statefun-playground. I can submit separate PR s those changes if everyone is on board. Also, should I plan to do the same upgrade to handle Flink 1.17.x? It should be easy to do, especially while the 1.16.x upgrade is fresh on my mind. Thanks. On Fri, Aug 11, 2023 at 6:40 PM Galen Warren <ga...@cvillewarrens.com> wrote: > I'm done with the code to make Statefun compatible with Flink 1.16, and > all the tests (including e2e succeed). The required changes were pretty > minimal. > > I'm running into a bit of a chicken/egg problem executing the tests in > flink-statefun-playground > <https://github.com/apache/flink-statefun-playground>, though. That > project seems to assume that all the various Statefun artifacts are built > and deployed to the various public repositories already. I've looked into > trying to redirect references to local artifacts; however, that's also > tricky since all the building occurs in Docker containers. > > Gordon, is there a trick to running the sample code in > flink-statefun-playground against yet-unreleased code that I'm missing? > > Thanks. > > On Sat, Jun 24, 2023 at 12:40 PM Galen Warren <ga...@cvillewarrens.com> > wrote: > >> Great -- thanks! >> >> I'm going to be out of town for about a week but I'll take a look at this >> when I'm back. >> >> On Tue, Jun 20, 2023 at 8:46 AM Martijn Visser <mvis...@confluent.io> >> wrote: >> >>> Hi Galen, >>> >>> Yes, I'll be more than happy to help with Statefun releases. >>> >>> Best regards, >>> >>> Martijn >>> >>> On Tue, Jun 20, 2023 at 2:21 PM Galen Warren <ga...@cvillewarrens.com> >>> wrote: >>> >>>> Thanks. >>>> >>>> Martijn, to answer your question, I'd need to do a small amount of work >>>> to get a PR ready, but not much. Happy to do it if we're deciding to >>>> restart Statefun releases -- are we? >>>> >>>> -- Galen >>>> >>>> On Sat, Jun 17, 2023 at 9:47 AM Tzu-Li (Gordon) Tai < >>>> tzuli...@apache.org> wrote: >>>> >>>>> > Perhaps he could weigh in on whether the combination of automated >>>>> tests plus those smoke tests should be sufficient for testing with new >>>>> Flink versions >>>>> >>>>> What we usually did at the bare minimum for new StateFun releases was >>>>> the following: >>>>> >>>>> 1. Build tests (including the smoke tests in the e2e module, which >>>>> covers important tests like exactly-once verification) >>>>> 2. Updating the flink-statefun-playground repo and manually >>>>> running all language examples there. >>>>> >>>>> If upgrading Flink versions was the only change in the release, I'd >>>>> probably say that this is sufficient. >>>>> >>>>> Best, >>>>> Gordon >>>>> >>>>> On Thu, Jun 15, 2023 at 5:25 AM Martijn Visser < >>>>> martijnvis...@apache.org> wrote: >>>>> >>>>>> Let me know if you have a PR for a Flink update :) >>>>>> >>>>>> On Thu, Jun 8, 2023 at 5:52 PM Galen Warren via user < >>>>>> u...@flink.apache.org> wrote: >>>>>> >>>>>>> Thanks Martijn. >>>>>>> >>>>>>> Personally, I'm already using a local fork of Statefun that is >>>>>>> compatible with Flink 1.16.x, so I wouldn't have any need for a released >>>>>>> version compatible with 1.15.x. I'd be happy to do the PRs to modify >>>>>>> Statefun to work with new versions of Flink as they come along. >>>>>>> >>>>>>> As for testing, Statefun does have unit tests and Gordon also sent >>>>>>> me instructions a while back for how to do some additional smoke tests >>>>>>> which are pretty straightforward. Perhaps he could weigh in on whether >>>>>>> the >>>>>>> combination of automated tests plus those smoke tests should be >>>>>>> sufficient >>>>>>> for testing with new Flink versions (I believe the answer is yes). >>>>>>> >>>>>>> -- Galen >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 8, 2023 at 8:01 AM Martijn Visser < >>>>>>> martijnvis...@apache.org> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Apologies for the late reply. >>>>>>>> >>>>>>>> I'm willing to help out with merging requests in Statefun to keep >>>>>>>> them >>>>>>>> compatible with new Flink releases and create new releases. I do >>>>>>>> think that >>>>>>>> validation of the functionality of these releases depends a lot on >>>>>>>> those >>>>>>>> who do these compatibility updates, with PMC members helping out >>>>>>>> with the >>>>>>>> formal process. >>>>>>>> >>>>>>>> > Why can't the Apache Software Foundation allow community members >>>>>>>> to bring >>>>>>>> it up to date? >>>>>>>> >>>>>>>> There's nothing preventing anyone from reviewing any of the current >>>>>>>> PRs or >>>>>>>> opening new ones. However, none of them are approved [1], so >>>>>>>> there's also >>>>>>>> nothing to merge. >>>>>>>> >>>>>>>> > I believe that there are people and companies on this mailing list >>>>>>>> interested in supporting Apache Flink Stateful Functions. >>>>>>>> >>>>>>>> If so, then now is the time to show. >>>>>>>> >>>>>>>> Would there be a preference to create a release with Galen's merged >>>>>>>> compatibility update to Flink 1.15.2, or do we want to skip that >>>>>>>> and go >>>>>>>> straight to a newer version? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Martijn >>>>>>>> >>>>>>>> [1] >>>>>>>> >>>>>>>> https://github.com/apache/flink-statefun/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved >>>>>>>> >>>>>>>> On Tue, Jun 6, 2023 at 3:55 PM Marco Villalobos < >>>>>>>> mvillalo...@kineteque.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> > Why can't the Apache Software Foundation allow community members >>>>>>>> to bring >>>>>>>> > it up to date? >>>>>>>> > >>>>>>>> > What's the process for that? >>>>>>>> > >>>>>>>> > I believe that there are people and companies on this mailing list >>>>>>>> > interested in supporting Apache Flink Stateful Functions. >>>>>>>> > >>>>>>>> > You already had two people on this thread express interest. >>>>>>>> > >>>>>>>> > At the very least, we could keep the library versions up to date. >>>>>>>> > >>>>>>>> > There are only a small list of new features that might be >>>>>>>> worthwhile: >>>>>>>> > >>>>>>>> > 1. event time processing >>>>>>>> > 2. state rest api >>>>>>>> > >>>>>>>> > >>>>>>>> > On Jun 6, 2023, at 3:06 AM, Chesnay Schepler <ches...@apache.org> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > If you were to fork it *and want to redistribute it* then the >>>>>>>> short >>>>>>>> > version is that >>>>>>>> > >>>>>>>> > 1. you have to adhere to the Apache licensing requirements >>>>>>>> > 2. you have to make it clear that your fork does not belong to >>>>>>>> the >>>>>>>> > Apache Flink project. (Trademarks and all that) >>>>>>>> > >>>>>>>> > Neither should be significant hurdles (there should also be >>>>>>>> plenty of >>>>>>>> > online resources regarding 1), and if you do this then you can >>>>>>>> freely share >>>>>>>> > your fork with others. >>>>>>>> > >>>>>>>> > I've also pinged Martijn to take a look at this thread. >>>>>>>> > To my knowledge the project hasn't decided anything yet. >>>>>>>> > >>>>>>>> > On 27/05/2023 04:05, Galen Warren wrote: >>>>>>>> > >>>>>>>> > Ok, I get it. No interest. >>>>>>>> > >>>>>>>> > If this project is being abandoned, I guess I'll work with my own >>>>>>>> fork. Is >>>>>>>> > there anything I should consider here? Can I share it with other >>>>>>>> people who >>>>>>>> > use this project? >>>>>>>> > >>>>>>>> > On Tue, May 16, 2023 at 10:50 AM Galen Warren < >>>>>>>> ga...@cvillewarrens.com> <ga...@cvillewarrens.com> >>>>>>>> > wrote: >>>>>>>> > >>>>>>>> > >>>>>>>> > Hi Martijn, since you opened this discussion thread, I'm curious >>>>>>>> what your >>>>>>>> > thoughts are in light of the responses? Thanks. >>>>>>>> > >>>>>>>> > On Wed, Apr 19, 2023 at 1:21 PM Galen Warren < >>>>>>>> ga...@cvillewarrens.com> <ga...@cvillewarrens.com> >>>>>>>> > wrote: >>>>>>>> > >>>>>>>> > >>>>>>>> > I use Apache Flink for stream processing, and StateFun as a >>>>>>>> hand-off >>>>>>>> > >>>>>>>> > point for the rest of the application. >>>>>>>> > It serves well as a bridge between a Flink Streaming job and >>>>>>>> > micro-services. >>>>>>>> > >>>>>>>> > This is essentially how I use it as well, and I would also be sad >>>>>>>> to see >>>>>>>> > it sunsetted. It works well; I don't know that there is a lot of >>>>>>>> new >>>>>>>> > development required, but if there are no new Statefun releases, >>>>>>>> then >>>>>>>> > Statefun can only be used with older Flink versions. >>>>>>>> > >>>>>>>> > On Tue, Apr 18, 2023 at 10:04 PM Marco Villalobos < >>>>>>>> mvillalo...@kineteque.com> wrote: >>>>>>>> > >>>>>>>> > >>>>>>>> > I am currently using Stateful Functions in my application. >>>>>>>> > >>>>>>>> > I use Apache Flink for stream processing, and StateFun as a >>>>>>>> hand-off >>>>>>>> > point for the rest of the application. >>>>>>>> > It serves well as a bridge between a Flink Streaming job and >>>>>>>> > micro-services. >>>>>>>> > >>>>>>>> > I would be disappointed if StateFun was sunsetted. Its a good >>>>>>>> idea. >>>>>>>> > >>>>>>>> > If there is anything I can do to help, as a contributor perhaps, >>>>>>>> please >>>>>>>> > let me know. >>>>>>>> > >>>>>>>> > >>>>>>>> > On Apr 3, 2023, at 2:02 AM, Martijn Visser < >>>>>>>> martijnvis...@apache.org> <martijnvis...@apache.org> >>>>>>>> > >>>>>>>> > wrote: >>>>>>>> > >>>>>>>> > Hi everyone, >>>>>>>> > >>>>>>>> > I want to open a discussion on the status of the Statefun Project >>>>>>>> [1] >>>>>>>> > >>>>>>>> > in Apache Flink. As you might have noticed, there hasn't been much >>>>>>>> > development over the past months in the Statefun repository [2]. >>>>>>>> There is >>>>>>>> > currently a lack of active contributors and committers who are >>>>>>>> able to help >>>>>>>> > with the maintenance of the project. >>>>>>>> > >>>>>>>> > In order to improve the situation, we need to solve the lack of >>>>>>>> > >>>>>>>> > committers and the lack of contributors. >>>>>>>> > >>>>>>>> > On the lack of committers: >>>>>>>> > >>>>>>>> > 1. Ideally, there are some of the current Flink committers who >>>>>>>> have >>>>>>>> > >>>>>>>> > the bandwidth and can help with reviewing PRs and merging them. >>>>>>>> > >>>>>>>> > 2. If that's not an option, it could be a consideration that >>>>>>>> current >>>>>>>> > >>>>>>>> > committers only approve and review PRs, that are approved by >>>>>>>> those who are >>>>>>>> > willing to contribute to Statefun and if the CI passes >>>>>>>> > >>>>>>>> > On the lack of contributors: >>>>>>>> > >>>>>>>> > 3. Next to having this discussion on the Dev and User mailing >>>>>>>> list, we >>>>>>>> > >>>>>>>> > can also create a blog with a call for new contributors on the >>>>>>>> Flink >>>>>>>> > project website, send out some tweets on the Flink / Statefun >>>>>>>> twitter >>>>>>>> > accounts, post messages on Slack etc. In that message, we would >>>>>>>> inform how >>>>>>>> > those that are interested in contributing can start and where >>>>>>>> they could >>>>>>>> > reach out for more information. >>>>>>>> > >>>>>>>> > There's also option 4. where a group of interested people would >>>>>>>> split >>>>>>>> > >>>>>>>> > Statefun from the Flink project and make it a separate top level >>>>>>>> project >>>>>>>> > under the Apache Flink umbrella (similar as recently has happened >>>>>>>> with >>>>>>>> > Flink Table Store, which has become Apache Paimon). >>>>>>>> > >>>>>>>> > If we see no improvements in the coming period, we should consider >>>>>>>> > >>>>>>>> > sunsetting Statefun and communicate that clearly to the users. >>>>>>>> > >>>>>>>> > I'm looking forward to your thoughts. >>>>>>>> > >>>>>>>> > Best regards, >>>>>>>> > >>>>>>>> > Martijn >>>>>>>> > >>>>>>>> > [1] >>>>>>>> https://nightlies.apache.org/flink/flink-statefun-docs-master/ < >>>>>>>> > >>>>>>>> > https://nightlies.apache.org/flink/flink-statefun-docs-master/> >>>>>>>> > >>>>>>>> > [2] https://github.com/apache/flink-statefun < >>>>>>>> > >>>>>>>> > https://github.com/apache/flink-statefun> >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>
