Hi Galen, > locally built code is copied into the build containers so that it can be accessed during the build.
That's exactly what we had been doing for release testing, yes. Sorry I missed that detail in my previous response. And yes, that sounds like a reasonable approach. If I understand you correctly, the workflow would become this: 1. Build the StateFun repo locally to install the snapshot artifact jars + have a local base StateFun image. 2. Run the playground in "local" mode, so that it uses the local base StateFun image + builds the playground code using copied artifact jars (instead of pulling from Maven). That looks good to me! Thanks, Gordon On Fri, Aug 18, 2023 at 11:33 AM Galen Warren <ga...@cvillewarrens.com.invalid> wrote: > Thanks. > > If you were to build a local image, as you suggest, how do you access that > image when building the playground images? All the compilation of > playground code happens inside containers, so local images on the host > aren't available in those containers. Unless I'm missing something? > > I've slightly reworked things such that the playground images can be run in > one of two modes -- the default mode, which works like before, and a > "local" mode where locally built code is copied into the build containers > so that it can be accessed during the build. It works fine, you just have > to define a couple of environment variables when running docker-compose to > specify default vs. local mode and what versions of Flink and Statefun > should be referenced, and then you can build a run the local examples > without any additional steps. Does that sound like a reasonable approach? > > > On Fri, Aug 18, 2023 at 2:17 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > > > Hi Galen, > > > > > Gordon, is there a trick to running the sample code in > > flink-statefun-playground against yet-unreleased code that I'm missing? > > > > You'd have to locally build an image from the release branch, with a > > temporary image version tag. Then, in the flink-statefun-playground, > change > > the image versions in the docker-compose files to use that locally built > > image. IIRC, that's what we have been doing in the past. Admittedly, it's > > pretty manual - I don't think the CI manages this workflow. > > > > Thanks, > > Gordon > > > > On Mon, Aug 14, 2023 at 10:42 AM Galen Warren <ga...@cvillewarrens.com> > > wrote: > > > > > I created a pull request for this: [FLINK-31619] Upgrade Stateful > > > Functions to Flink 1.16.1 by galenwarren · Pull Request #331 · > > > apache/flink-statefun (github.com) > > > <https://github.com/apache/flink-statefun/pull/331>. > > > > > > JIRA is here: [FLINK-31619] Upgrade Stateful Functions to Flink 1.16.1 > - > > > ASF JIRA (apache.org) > > > <https://issues.apache.org/jira/browse/FLINK-31619?filter=-1>. > > > > > > Statefun references 1.16.2, despite the title -- that version has come > > out > > > since the issue was created. > > > > > > I figured out how to run all the playground tests locally, but it took > a > > > bit of reworking of the playground setup with respect to Docker; > > > specifically, the Docker contexts used to build the example functions > > > needed to be broadened (i.e. moved up the tree) so that, if needed, > local > > > artifacts/code can be accessed from within the containers at build > time. > > > Then I made the Docker compose.yml configurable through environment > > > variables to allow for them to run in either the original manner -- > i.e. > > > pulling artifacts from public repos -- or in a "local" mode, where > > > artifacts are pulled from local builds. > > > > > > This process is a cleaner if the playground is a subfolder of the > > > flink-statefun project rather than be its own repository > > > (flink-statefun-playground), because then all the relative paths > between > > > the playground files and the build artifacts are fixed. So, I'd like to > > > propose to move the playground files, modified as described above, to > > > flink-statefun/playground and retire flink-statefun-playground. I can > > > submit separate PR s those changes if everyone is on board. > > > > > > Also, should I plan to do the same upgrade to handle Flink 1.17.x? It > > > should be easy to do, especially while the 1.16.x upgrade is fresh on > my > > > mind. > > > > > > Thanks. > > > > > > > > > On Fri, Aug 11, 2023 at 6:40 PM Galen Warren <ga...@cvillewarrens.com> > > > wrote: > > > > > >> I'm done with the code to make Statefun compatible with Flink 1.16, > and > > >> all the tests (including e2e succeed). The required changes were > pretty > > >> minimal. > > >> > > >> I'm running into a bit of a chicken/egg problem executing the tests in > > >> flink-statefun-playground > > >> <https://github.com/apache/flink-statefun-playground>, though. That > > >> project seems to assume that all the various Statefun artifacts are > > built > > >> and deployed to the various public repositories already. I've looked > > into > > >> trying to redirect references to local artifacts; however, that's also > > >> tricky since all the building occurs in Docker containers. > > >> > > >> Gordon, is there a trick to running the sample code in > > >> flink-statefun-playground against yet-unreleased code that I'm > missing? > > >> > > >> Thanks. > > >> > > >> On Sat, Jun 24, 2023 at 12:40 PM Galen Warren < > ga...@cvillewarrens.com> > > >> wrote: > > >> > > >>> Great -- thanks! > > >>> > > >>> I'm going to be out of town for about a week but I'll take a look at > > >>> this when I'm back. > > >>> > > >>> On Tue, Jun 20, 2023 at 8:46 AM Martijn Visser <mvis...@confluent.io > > > > >>> wrote: > > >>> > > >>>> Hi Galen, > > >>>> > > >>>> Yes, I'll be more than happy to help with Statefun releases. > > >>>> > > >>>> Best regards, > > >>>> > > >>>> Martijn > > >>>> > > >>>> On Tue, Jun 20, 2023 at 2:21 PM Galen Warren < > ga...@cvillewarrens.com > > > > > >>>> wrote: > > >>>> > > >>>>> Thanks. > > >>>>> > > >>>>> Martijn, to answer your question, I'd need to do a small amount of > > >>>>> work to get a PR ready, but not much. Happy to do it if we're > > deciding to > > >>>>> restart Statefun releases -- are we? > > >>>>> > > >>>>> -- Galen > > >>>>> > > >>>>> On Sat, Jun 17, 2023 at 9:47 AM Tzu-Li (Gordon) Tai < > > >>>>> tzuli...@apache.org> wrote: > > >>>>> > > >>>>>> > Perhaps he could weigh in on whether the combination of > automated > > >>>>>> tests plus those smoke tests should be sufficient for testing with > > new > > >>>>>> Flink versions > > >>>>>> > > >>>>>> What we usually did at the bare minimum for new StateFun releases > > was > > >>>>>> the following: > > >>>>>> > > >>>>>> 1. Build tests (including the smoke tests in the e2e module, > > >>>>>> which covers important tests like exactly-once verification) > > >>>>>> 2. Updating the flink-statefun-playground repo and manually > > >>>>>> running all language examples there. > > >>>>>> > > >>>>>> If upgrading Flink versions was the only change in the release, > I'd > > >>>>>> probably say that this is sufficient. > > >>>>>> > > >>>>>> Best, > > >>>>>> Gordon > > >>>>>> > > >>>>>> On Thu, Jun 15, 2023 at 5:25 AM Martijn Visser < > > >>>>>> martijnvis...@apache.org> wrote: > > >>>>>> > > >>>>>>> Let me know if you have a PR for a Flink update :) > > >>>>>>> > > >>>>>>> On Thu, Jun 8, 2023 at 5:52 PM Galen Warren via user < > > >>>>>>> u...@flink.apache.org> wrote: > > >>>>>>> > > >>>>>>>> Thanks Martijn. > > >>>>>>>> > > >>>>>>>> Personally, I'm already using a local fork of Statefun that is > > >>>>>>>> compatible with Flink 1.16.x, so I wouldn't have any need for a > > released > > >>>>>>>> version compatible with 1.15.x. I'd be happy to do the PRs to > > modify > > >>>>>>>> Statefun to work with new versions of Flink as they come along. > > >>>>>>>> > > >>>>>>>> As for testing, Statefun does have unit tests and Gordon also > sent > > >>>>>>>> me instructions a while back for how to do some additional smoke > > tests > > >>>>>>>> which are pretty straightforward. Perhaps he could weigh in on > > whether the > > >>>>>>>> combination of automated tests plus those smoke tests should be > > sufficient > > >>>>>>>> for testing with new Flink versions (I believe the answer is > yes). > > >>>>>>>> > > >>>>>>>> -- Galen > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Thu, Jun 8, 2023 at 8:01 AM Martijn Visser < > > >>>>>>>> martijnvis...@apache.org> wrote: > > >>>>>>>> > > >>>>>>>>> Hi all, > > >>>>>>>>> > > >>>>>>>>> Apologies for the late reply. > > >>>>>>>>> > > >>>>>>>>> I'm willing to help out with merging requests in Statefun to > keep > > >>>>>>>>> them > > >>>>>>>>> compatible with new Flink releases and create new releases. I > do > > >>>>>>>>> think that > > >>>>>>>>> validation of the functionality of these releases depends a lot > > on > > >>>>>>>>> those > > >>>>>>>>> who do these compatibility updates, with PMC members helping > out > > >>>>>>>>> with the > > >>>>>>>>> formal process. > > >>>>>>>>> > > >>>>>>>>> > Why can't the Apache Software Foundation allow community > > members > > >>>>>>>>> to bring > > >>>>>>>>> it up to date? > > >>>>>>>>> > > >>>>>>>>> There's nothing preventing anyone from reviewing any of the > > >>>>>>>>> current PRs or > > >>>>>>>>> opening new ones. However, none of them are approved [1], so > > >>>>>>>>> there's also > > >>>>>>>>> nothing to merge. > > >>>>>>>>> > > >>>>>>>>> > I believe that there are people and companies on this mailing > > >>>>>>>>> list > > >>>>>>>>> interested in supporting Apache Flink Stateful Functions. > > >>>>>>>>> > > >>>>>>>>> If so, then now is the time to show. > > >>>>>>>>> > > >>>>>>>>> Would there be a preference to create a release with Galen's > > merged > > >>>>>>>>> compatibility update to Flink 1.15.2, or do we want to skip > that > > >>>>>>>>> and go > > >>>>>>>>> straight to a newer version? > > >>>>>>>>> > > >>>>>>>>> Best regards, > > >>>>>>>>> > > >>>>>>>>> Martijn > > >>>>>>>>> > > >>>>>>>>> [1] > > >>>>>>>>> > > >>>>>>>>> > > > https://github.com/apache/flink-statefun/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved > > >>>>>>>>> > > >>>>>>>>> On Tue, Jun 6, 2023 at 3:55 PM Marco Villalobos < > > >>>>>>>>> mvillalo...@kineteque.com> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> > Why can't the Apache Software Foundation allow community > > members > > >>>>>>>>> to bring > > >>>>>>>>> > it up to date? > > >>>>>>>>> > > > >>>>>>>>> > What's the process for that? > > >>>>>>>>> > > > >>>>>>>>> > I believe that there are people and companies on this mailing > > >>>>>>>>> list > > >>>>>>>>> > interested in supporting Apache Flink Stateful Functions. > > >>>>>>>>> > > > >>>>>>>>> > You already had two people on this thread express interest. > > >>>>>>>>> > > > >>>>>>>>> > At the very least, we could keep the library versions up to > > date. > > >>>>>>>>> > > > >>>>>>>>> > There are only a small list of new features that might be > > >>>>>>>>> worthwhile: > > >>>>>>>>> > > > >>>>>>>>> > 1. event time processing > > >>>>>>>>> > 2. state rest api > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > On Jun 6, 2023, at 3:06 AM, Chesnay Schepler < > > ches...@apache.org> > > >>>>>>>>> wrote: > > >>>>>>>>> > > > >>>>>>>>> > If you were to fork it *and want to redistribute it* then the > > >>>>>>>>> short > > >>>>>>>>> > version is that > > >>>>>>>>> > > > >>>>>>>>> > 1. you have to adhere to the Apache licensing requirements > > >>>>>>>>> > 2. you have to make it clear that your fork does not > belong > > >>>>>>>>> to the > > >>>>>>>>> > Apache Flink project. (Trademarks and all that) > > >>>>>>>>> > > > >>>>>>>>> > Neither should be significant hurdles (there should also be > > >>>>>>>>> plenty of > > >>>>>>>>> > online resources regarding 1), and if you do this then you > can > > >>>>>>>>> freely share > > >>>>>>>>> > your fork with others. > > >>>>>>>>> > > > >>>>>>>>> > I've also pinged Martijn to take a look at this thread. > > >>>>>>>>> > To my knowledge the project hasn't decided anything yet. > > >>>>>>>>> > > > >>>>>>>>> > On 27/05/2023 04:05, Galen Warren wrote: > > >>>>>>>>> > > > >>>>>>>>> > Ok, I get it. No interest. > > >>>>>>>>> > > > >>>>>>>>> > If this project is being abandoned, I guess I'll work with my > > >>>>>>>>> own fork. Is > > >>>>>>>>> > there anything I should consider here? Can I share it with > > other > > >>>>>>>>> people who > > >>>>>>>>> > use this project? > > >>>>>>>>> > > > >>>>>>>>> > On Tue, May 16, 2023 at 10:50 AM Galen Warren < > > >>>>>>>>> ga...@cvillewarrens.com> <ga...@cvillewarrens.com> > > >>>>>>>>> > wrote: > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > Hi Martijn, since you opened this discussion thread, I'm > > curious > > >>>>>>>>> what your > > >>>>>>>>> > thoughts are in light of the responses? Thanks. > > >>>>>>>>> > > > >>>>>>>>> > On Wed, Apr 19, 2023 at 1:21 PM Galen Warren < > > >>>>>>>>> ga...@cvillewarrens.com> <ga...@cvillewarrens.com> > > >>>>>>>>> > wrote: > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > I use Apache Flink for stream processing, and StateFun as a > > >>>>>>>>> hand-off > > >>>>>>>>> > > > >>>>>>>>> > point for the rest of the application. > > >>>>>>>>> > It serves well as a bridge between a Flink Streaming job and > > >>>>>>>>> > micro-services. > > >>>>>>>>> > > > >>>>>>>>> > This is essentially how I use it as well, and I would also be > > >>>>>>>>> sad to see > > >>>>>>>>> > it sunsetted. It works well; I don't know that there is a lot > > of > > >>>>>>>>> new > > >>>>>>>>> > development required, but if there are no new Statefun > > releases, > > >>>>>>>>> then > > >>>>>>>>> > Statefun can only be used with older Flink versions. > > >>>>>>>>> > > > >>>>>>>>> > On Tue, Apr 18, 2023 at 10:04 PM Marco Villalobos < > > >>>>>>>>> mvillalo...@kineteque.com> wrote: > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > I am currently using Stateful Functions in my application. > > >>>>>>>>> > > > >>>>>>>>> > I use Apache Flink for stream processing, and StateFun as a > > >>>>>>>>> hand-off > > >>>>>>>>> > point for the rest of the application. > > >>>>>>>>> > It serves well as a bridge between a Flink Streaming job and > > >>>>>>>>> > micro-services. > > >>>>>>>>> > > > >>>>>>>>> > I would be disappointed if StateFun was sunsetted. Its a > good > > >>>>>>>>> idea. > > >>>>>>>>> > > > >>>>>>>>> > If there is anything I can do to help, as a contributor > > perhaps, > > >>>>>>>>> please > > >>>>>>>>> > let me know. > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > On Apr 3, 2023, at 2:02 AM, Martijn Visser < > > >>>>>>>>> martijnvis...@apache.org> <martijnvis...@apache.org> > > >>>>>>>>> > > > >>>>>>>>> > wrote: > > >>>>>>>>> > > > >>>>>>>>> > Hi everyone, > > >>>>>>>>> > > > >>>>>>>>> > I want to open a discussion on the status of the Statefun > > >>>>>>>>> Project [1] > > >>>>>>>>> > > > >>>>>>>>> > in Apache Flink. As you might have noticed, there hasn't been > > >>>>>>>>> much > > >>>>>>>>> > development over the past months in the Statefun repository > > [2]. > > >>>>>>>>> There is > > >>>>>>>>> > currently a lack of active contributors and committers who > are > > >>>>>>>>> able to help > > >>>>>>>>> > with the maintenance of the project. > > >>>>>>>>> > > > >>>>>>>>> > In order to improve the situation, we need to solve the lack > of > > >>>>>>>>> > > > >>>>>>>>> > committers and the lack of contributors. > > >>>>>>>>> > > > >>>>>>>>> > On the lack of committers: > > >>>>>>>>> > > > >>>>>>>>> > 1. Ideally, there are some of the current Flink committers > who > > >>>>>>>>> have > > >>>>>>>>> > > > >>>>>>>>> > the bandwidth and can help with reviewing PRs and merging > them. > > >>>>>>>>> > > > >>>>>>>>> > 2. If that's not an option, it could be a consideration that > > >>>>>>>>> current > > >>>>>>>>> > > > >>>>>>>>> > committers only approve and review PRs, that are approved by > > >>>>>>>>> those who are > > >>>>>>>>> > willing to contribute to Statefun and if the CI passes > > >>>>>>>>> > > > >>>>>>>>> > On the lack of contributors: > > >>>>>>>>> > > > >>>>>>>>> > 3. Next to having this discussion on the Dev and User mailing > > >>>>>>>>> list, we > > >>>>>>>>> > > > >>>>>>>>> > can also create a blog with a call for new contributors on > the > > >>>>>>>>> Flink > > >>>>>>>>> > project website, send out some tweets on the Flink / Statefun > > >>>>>>>>> twitter > > >>>>>>>>> > accounts, post messages on Slack etc. In that message, we > would > > >>>>>>>>> inform how > > >>>>>>>>> > those that are interested in contributing can start and where > > >>>>>>>>> they could > > >>>>>>>>> > reach out for more information. > > >>>>>>>>> > > > >>>>>>>>> > There's also option 4. where a group of interested people > would > > >>>>>>>>> split > > >>>>>>>>> > > > >>>>>>>>> > Statefun from the Flink project and make it a separate top > > level > > >>>>>>>>> project > > >>>>>>>>> > under the Apache Flink umbrella (similar as recently has > > >>>>>>>>> happened with > > >>>>>>>>> > Flink Table Store, which has become Apache Paimon). > > >>>>>>>>> > > > >>>>>>>>> > If we see no improvements in the coming period, we should > > >>>>>>>>> consider > > >>>>>>>>> > > > >>>>>>>>> > sunsetting Statefun and communicate that clearly to the > users. > > >>>>>>>>> > > > >>>>>>>>> > I'm looking forward to your thoughts. > > >>>>>>>>> > > > >>>>>>>>> > Best regards, > > >>>>>>>>> > > > >>>>>>>>> > Martijn > > >>>>>>>>> > > > >>>>>>>>> > [1] > > >>>>>>>>> https://nightlies.apache.org/flink/flink-statefun-docs-master/ > < > > >>>>>>>>> > > > >>>>>>>>> > > https://nightlies.apache.org/flink/flink-statefun-docs-master/ > > > > > >>>>>>>>> > > > >>>>>>>>> > [2] https://github.com/apache/flink-statefun < > > >>>>>>>>> > > > >>>>>>>>> > https://github.com/apache/flink-statefun> > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>> > > >
