[NOTICE] Experimental Java 17 support now available on master
I built the Flink master branch and tried running this simple Flink app that uses a Java record: https://github.com/kurtostfeld/flink-kryo-upgrade-demo/blob/main/flink-record-demo/src/main/java/demo/app/Main.java It fails with the normal exception that Kryo 2.x throws when you try to serialize a Java record. The full stack trace is here: https://pastebin.com/HGhGKUWt I tried removing this line: https://github.com/kurtostfeld/flink-kryo-upgrade-demo/blob/main/flink-record-demo/src/main/java/demo/app/Main.java#L36 and that had no impact, I got the same error. In the other thread, you said that the plan was to use PojoSerializer to serialize records rather than Kryo. Currently, the Flink code bases uses Kryo 2.x by default for generic user data types, and that will fail when the data type is a record or contains records. Ultimately, if Flink wants to fully support Java records, it seems that it has to move off of Kryo 2.x. PojoSerializer is part of what is basically a custom serialization library internal to Flink that is an alternative to Kryo. That's one option: move off of Kryo to a Flink-internal serialization library. The other two options are upgrade to the new Kryo or use a different serialization library. The Kryo 5.5.0 upgrade PR I submitted (https://github.com/apache/flink/pull/22660) with FLIP 317 (https://cwiki.apache.org/confluence/display/FLINK/FLIP-317%3A+Upgrade+Kryo+from+2.24.0+to+5.5.0) works with records. The Flink app linked above that uses records works with the PR and that's what I posted to this mailing list a few weeks ago. I rebased the pull request on to the latest master branch and it's passing all tests. From my testing, it supports stateful upgrades, including checkpoints. If you can demonstrate a scenario where stateful upgrades error I can try to resolve that.
Re: [DISCUSS] FLIP-321: Introduce an API deprecation process
Hi Becket, Thanks for this FLIP! Having a deprecation process is really important. I understand some people’s concerns about the additional burden for project maintainers, but my personal experience with Kafka has been that it’s very liveable and that it’s well worth the benefit to users. In fact, users being able to confidently upgrade is also a benefit to maintainers, as we will get fewer questions from people stuck on very old versions. One question: Can we consider only dropping deprecated APIs in major releases across the board? I understand that Experimental and PublicEvolving APIs are by definition less stable, but it seems like this should be reflected in the required deprecation period alone. I.e. that we must keep them around for at least zero or one minor release, not that we can drop them in a minor or patch release. The advantage of forbidding the removal of any API in minor or patch releases is that users will get a strong guarantee that they can bump the minor or patch version and still be able to compile, or even just re-link and know that they won’t face “MethodDef” exceptions at run time. This is a binary guarantee: if we allow removing even Experimental APIs outside of major releases, users can no longer confidently upgrade. Aside from that, I’d share my 2 cents on a couple of points: * I’d use the official Deprecated annotation instead of introducing our own flavor (Retired, etc), since Deprecated is well integrated into build tools and IDEs. * I wouldn’t worry about a demotion process in this FLIP; it seems orthogonal, and something that should probably be taken case-by-case anyway. * Aside from deprecation and removal, there have been some discussions about how to evolve APIs and behavior in compatible ways. This is somewhat of an art, and if folks haven’t wrestled with it before, it’ll take some time to become good at it. I feel like this topic should also be orthogonal to this FLIP, but FWIW, my suggestion would be to adopt a simple policy not to break existing user programs, and leave the “how” up to implementers and reviewers. Thanks again, John On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: > Hi All, > > The @Public -> @PublicEvolving proposed by Xintong is a great idea. > Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2 > minor release)--> @Public --> @deprecated --(1 major > release)--> @PublicRetired. It will provide a lot of flexibility without > breaking any rules we had. @Public APIs are allowed to change between major > releases. Changing annotations is acceptable and provides additional > tolerance i.e. user-friendliness, since the APIs themself are not changed. > > I had similar thoughts when I was facing those issues. I want to move one > step further and suggest introducing one more annotation @Retired. > > Not like the @PublicRetired which is a compromise of downgrading @Public to > @PublicEvolving. As I mentioned earlier in my reply, Java standard > @deprecated should be used in the early stage of the deprecation process > and doesn't really meet our requirement. Since Java does not allow us to > extend annotation, I think it would be feasible to have the new @Retired to > help us monitor and manage the deprecation process, house cleaning, etc. > > Some ideas could be(open for discussion): > > @Retired: > > 1. There must be a replacement with functionality compatibility before APIs > can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are > mandatory (a FLIP is recommended). > 2. APIs marked as @Retired will be removed after 1 minor release sharply > (using ArchUnit to force it, needs further check whether it is possible). > Devs who marked them as @Retired are responsible to remove them. > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are > recommended. @Experimental -> @Retired and @Internal -> @Retired could also > be used if it can increase user-friendliness or dev-friendliness, but not > mandatory. > 4. Some variables will be defined in @Retired to support the deprecation > process management. Further extension is possible, since the annotation is > built by us. > > > Best regards, > Jing > > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin wrote: > >> Hi Xintong, >> >> Thanks for the explanation. Please see the replies inline below. >> >> I agree. And from my understanding, demoting a Public API is also a kind of >> > such change, just like removing one, which can only happen with major >> > version bumps. I'm not proposing to allow demoting Public APIs anytime, >> but >> > only in the case major version bumps happen before reaching the >> > 2-minor-release migration period. Actually, demoting would be a weaker >> > change compared to removing the API immediately upon major version bumps, >> > in order to keep the commitment about the 2-minor-release migration >> period. >> > If the concern is that `@Public` -> `@PublicEvolving` sounds against >> > conventions, we may introduce a
Re: [DISCUSS] FLIP-321: Introduce an API deprecation process
Hi All, The @Public -> @PublicEvolving proposed by Xintong is a great idea. Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2 minor release)--> @Public --> @deprecated --(1 major release)--> @PublicRetired. It will provide a lot of flexibility without breaking any rules we had. @Public APIs are allowed to change between major releases. Changing annotations is acceptable and provides additional tolerance i.e. user-friendliness, since the APIs themself are not changed. I had similar thoughts when I was facing those issues. I want to move one step further and suggest introducing one more annotation @Retired. Not like the @PublicRetired which is a compromise of downgrading @Public to @PublicEvolving. As I mentioned earlier in my reply, Java standard @deprecated should be used in the early stage of the deprecation process and doesn't really meet our requirement. Since Java does not allow us to extend annotation, I think it would be feasible to have the new @Retired to help us monitor and manage the deprecation process, house cleaning, etc. Some ideas could be(open for discussion): @Retired: 1. There must be a replacement with functionality compatibility before APIs can be marked as @Retired, i.e. DISCUSS and VOTE processes on the ML are mandatory (a FLIP is recommended). 2. APIs marked as @Retired will be removed after 1 minor release sharply (using ArchUnit to force it, needs further check whether it is possible). Devs who marked them as @Retired are responsible to remove them. 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are recommended. @Experimental -> @Retired and @Internal -> @Retired could also be used if it can increase user-friendliness or dev-friendliness, but not mandatory. 4. Some variables will be defined in @Retired to support the deprecation process management. Further extension is possible, since the annotation is built by us. Best regards, Jing On Fri, Jun 16, 2023 at 10:31 AM Becket Qin wrote: > Hi Xintong, > > Thanks for the explanation. Please see the replies inline below. > > I agree. And from my understanding, demoting a Public API is also a kind of > > such change, just like removing one, which can only happen with major > > version bumps. I'm not proposing to allow demoting Public APIs anytime, > but > > only in the case major version bumps happen before reaching the > > 2-minor-release migration period. Actually, demoting would be a weaker > > change compared to removing the API immediately upon major version bumps, > > in order to keep the commitment about the 2-minor-release migration > period. > > If the concern is that `@Public` -> `@PublicEvolving` sounds against > > conventions, we may introduce a new annotation if necessary, e.g., > > `@PublicRetiring`, to avoid confusions. > > As an end user who only uses Public APIs, if I don't change my code at all, > my expectation is the following: > 1. Upgrading from 1.x to 2.x may have issues. > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with all > the 2.x versions. > Actually I think there are some dependency version resolution policies out > there which picks the highest minor version when the dependencies pull in > multiple minor versions of the same jar, which may be broken if we remove > the API in minor releases. > > I'm not sure about this. Yes, it's completely "legal" that we bump up the > > major version whenever a breaking change is needed. However, this also > > weakens the value of the commitment that public APIs will stay stable > > within the major release series, as the series can end anytime. IMHO, > short > > major release series are not something "make the end users happy", but > > backdoors that allow us as the developers to make frequent breaking > > changes. On the contrary, with the demoting approach, we can still have > > longer major release series, while only allowing Public APIs deprecated > at > > the end of the previous major version to be removed in the next major > > version. > > I totally agree that frequent major version bumps are not ideal, but here > we are comparing it with a minor version bump which removes a Public API. > So the context is that we have already decided to remove this Public API > while keeping everything else backwards compatible. I think a major version > bump is a commonly understood signal here, compared with a minor version > change. From end users' perspective, for those who are not impacted, in > this case upgrading a major version is not necessarily more involved than > upgrading a minor version - both should be as smooth as a dependency > version change. For those who are impacted, they will lose the Public API > anyways and a major version bump ensures there is no surprise. > > Thanks, > > Jiangjie (Becket) Qin > > On Fri, Jun 16, 2023 at 10:13 AM Xintong Song > wrote: > > > Public API is a well defined common concept, and one of its > >> convention is that it only changes with a major version change. > >> > >
Re: [DISCUSS] FLIP-321: Introduce an API deprecation process
Hi Becket, Thanks for the clarification. Sorry that I didn't make myself clear enough. Let me share more thoughts. > Deprecating an API is just a more elegant way of replacing an API with a > new one. The only difference between the two is whether the old API is kept > and coexists with the new API for some releases or not. For end users, > deprecation-then-remove is much more friendly than direct replacement. > Making old experimental API @deprecated with new API as replacement needs extra effort. Let's use the example in FLIP to see what happened. Deprecation rule A: - API foo was introduced in 1.18.x - API foo was carried over in 1.19.0 - API foo was marked as @deprecated in 1.19.1 - API foo is moved in 1.19.2 - the breaking change Deprecation rule B: - API foo was introduced in 1.18.x - API foo was carried over in 1.19.0 - API foo was moved in 1.19.1 - the breaking change - Another release 1.19.2 1.19.2 is the current/last release. One key concern of users is to limit the effort as much as possible for any patch upgrade. Let's name it "the rule of patch upgrade effort". If this is not true, I'd like to know more different thoughts. There will be following scenarios: Scenarios 1: Following the rule of patch upgrade effort: 1. upgrade from 1.18.x to 1.19.x. This is an upgrade across minor releases. Users will upgrade to 1.19.2(the last release), since there will be many bigger changes than the foo-bar change. There is no difference between deprecation rule A and B. 2. upgrade from 1.19.0 to 1.19.x. To minimize the upgrade effort, with the deprecation rule A, users will upgrade to 1.19.1 while with the deprecation rule B, users will stay with 1.19.0 and keep waiting for the next minor release to handle breaking changes with effort. Since the last release is 1.19.2, users are using an old release in both cases. The deprecation rule A is only slightly better, when there are other bug-fixes in 1.19.1 that are urgently required for them. In this case, we will go to the second scenario. Scenarios 2: Don't follow the rule of patch upgrade effort: 1. upgrade from 1.18.x to 1.19.x. Same as in scenario 1. This is an upgrade across minor releases. Users will upgrade to 1.19.2(the last release). There is no difference between deprecation rule A and B. 2. upgrade from 1.19.0 to 1.19.x. Focus will be on upgrading to the last version instead of effort. With both the deprecation rule A and B, users will handle the breaking change and upgrade to 1.19.2. There is no difference between them. As we can see, with the (big) effort of maintaining old experimental APIs, the value is very limited. And in scenario 1.2, since the new patch release 1.19.1 contains urgent bug-fixes, it is acceptable for those users who really need them to put some effort into upgrading (Deprecation rule B). After all, everyone was aware that @experimental can be changed between patch releases in the first place. This is orthogonal to the deprecation process, and may not even be required > in some cases if the function changes by itself. But in general, this > relies on the developer to decide. A simple test on readiness is to see if > all the UT / IT cases written with the old API can be migrated to the new > one and still work. If the new API is not ready, we probably should not > deprecate the old one to begin with. > I have a different thought. My gut feeling is that, once this FLIP is done, developers will use it as the golden rule each time when deprecated APIs need to be removed. For complex features, the deprecation phase will take time, Marking old APIs deprecated as early as possible will give users more time to migrate or give them the heads-up earlier. It is more comfortable for users than having everything read and marking old APIs as deprecated sharply and removing them after a defined migration period. Because the longer we let such abandoned APIs not be marked as deprecated, the more dependencies users will be building with them, and the bigger effort for users to do migration in the future. > By "backwards compatible", do you mean functionally equivalent? If the new > APIs are designed to be not backwards compatible, then removing the > deprecated source code is definitely allowed. If we don't think the new API > is ready to take over the place for the old one, then we should wait. The > migration period is the minimum time we have to wait before removing the > source code. A longer migration period is OK. > > Yes, by "backwards compatible", I meant functionally equivalent. You can find more details in this thread[1]. I should have used "functionally compatible" instead. Sorry for the confusion. This is also the example I want to point out for the thought described above. If the FLIP does not contain enough information, guidelines, conditions, the migration period will be the only rule to decide when to remove deprecated APIs without caring of anything else like functionality compatibility. If the migration
Re: [DISCUSS] Status of Statefun Project
> Perhaps he could weigh in on whether the combination of automated tests plus those smoke tests should be sufficient for testing with new Flink versions What we usually did at the bare minimum for new StateFun releases was the following: 1. Build tests (including the smoke tests in the e2e module, which covers important tests like exactly-once verification) 2. Updating the flink-statefun-playground repo and manually running all language examples there. If upgrading Flink versions was the only change in the release, I'd probably say that this is sufficient. Best, Gordon On Thu, Jun 15, 2023 at 5:25 AM Martijn Visser wrote: > Let me know if you have a PR for a Flink update :) > > On Thu, Jun 8, 2023 at 5:52 PM Galen Warren via user < > u...@flink.apache.org> wrote: > >> Thanks Martijn. >> >> Personally, I'm already using a local fork of Statefun that is compatible >> with Flink 1.16.x, so I wouldn't have any need for a released version >> compatible with 1.15.x. I'd be happy to do the PRs to modify Statefun to >> work with new versions of Flink as they come along. >> >> As for testing, Statefun does have unit tests and Gordon also sent me >> instructions a while back for how to do some additional smoke tests which >> are pretty straightforward. Perhaps he could weigh in on whether the >> combination of automated tests plus those smoke tests should be sufficient >> for testing with new Flink versions (I believe the answer is yes). >> >> -- Galen >> >> >> >> On Thu, Jun 8, 2023 at 8:01 AM Martijn Visser >> wrote: >> >>> Hi all, >>> >>> Apologies for the late reply. >>> >>> I'm willing to help out with merging requests in Statefun to keep them >>> compatible with new Flink releases and create new releases. I do think >>> that >>> validation of the functionality of these releases depends a lot on those >>> who do these compatibility updates, with PMC members helping out with the >>> formal process. >>> >>> > Why can't the Apache Software Foundation allow community members to >>> bring >>> it up to date? >>> >>> There's nothing preventing anyone from reviewing any of the current PRs >>> or >>> opening new ones. However, none of them are approved [1], so there's also >>> nothing to merge. >>> >>> > I believe that there are people and companies on this mailing list >>> interested in supporting Apache Flink Stateful Functions. >>> >>> If so, then now is the time to show. >>> >>> Would there be a preference to create a release with Galen's merged >>> compatibility update to Flink 1.15.2, or do we want to skip that and go >>> straight to a newer version? >>> >>> Best regards, >>> >>> Martijn >>> >>> [1] >>> >>> https://github.com/apache/flink-statefun/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved >>> >>> On Tue, Jun 6, 2023 at 3:55 PM Marco Villalobos < >>> mvillalo...@kineteque.com> >>> wrote: >>> >>> > Why can't the Apache Software Foundation allow community members to >>> bring >>> > it up to date? >>> > >>> > What's the process for that? >>> > >>> > I believe that there are people and companies on this mailing list >>> > interested in supporting Apache Flink Stateful Functions. >>> > >>> > You already had two people on this thread express interest. >>> > >>> > At the very least, we could keep the library versions up to date. >>> > >>> > There are only a small list of new features that might be worthwhile: >>> > >>> > 1. event time processing >>> > 2. state rest api >>> > >>> > >>> > On Jun 6, 2023, at 3:06 AM, Chesnay Schepler >>> wrote: >>> > >>> > If you were to fork it *and want to redistribute it* then the short >>> > version is that >>> > >>> >1. you have to adhere to the Apache licensing requirements >>> >2. you have to make it clear that your fork does not belong to the >>> >Apache Flink project. (Trademarks and all that) >>> > >>> > Neither should be significant hurdles (there should also be plenty of >>> > online resources regarding 1), and if you do this then you can freely >>> share >>> > your fork with others. >>> > >>> > I've also pinged Martijn to take a look at this thread. >>> > To my knowledge the project hasn't decided anything yet. >>> > >>> > On 27/05/2023 04:05, Galen Warren wrote: >>> > >>> > Ok, I get it. No interest. >>> > >>> > If this project is being abandoned, I guess I'll work with my own >>> fork. Is >>> > there anything I should consider here? Can I share it with other >>> people who >>> > use this project? >>> > >>> > On Tue, May 16, 2023 at 10:50 AM Galen Warren >>> >>> > wrote: >>> > >>> > >>> > Hi Martijn, since you opened this discussion thread, I'm curious what >>> your >>> > thoughts are in light of the responses? Thanks. >>> > >>> > On Wed, Apr 19, 2023 at 1:21 PM Galen Warren >>> >>> > wrote: >>> > >>> > >>> > I use Apache Flink for stream processing, and StateFun as a hand-off >>> > >>> > point for the rest of the application. >>> > It serves well as a bridge between a Flink Streaming job and >>> > micro-services. >>> > >>> > This is
[jira] [Created] (FLINK-32373) Support passing headers with SQL Client gateway requests
Alexander Fedulov created FLINK-32373: - Summary: Support passing headers with SQL Client gateway requests Key: FLINK-32373 URL: https://issues.apache.org/jira/browse/FLINK-32373 Project: Flink Issue Type: Improvement Components: Table SQL / Client, Table SQL / Gateway Affects Versions: 1.18.0 Reporter: Alexander Fedulov FLINK-32030 and FLINK-32035 enable communication from the SQL Client to the SQL Gateway placed behind a proxy, such as a K8S ingress. Given that authentication is typically needed in these cases, it can be achieved by adding the ability to supply custom headers to the underlying RestClient. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32372) flink-connector-aws: build on pull request / compile_and_test doesn't support for Flink 1.16.2 and 1.17.1
Samrat Deb created FLINK-32372:
--
Summary: flink-connector-aws: build on pull request /
compile_and_test doesn't support for Flink 1.16.2 and 1.17.1
Key: FLINK-32372
URL: https://issues.apache.org/jira/browse/FLINK-32372
Project: Flink
Issue Type: New Feature
Components: Connectors / AWS
Reporter: Samrat Deb
*Recently* {*}1.16.2 and 1.17.1 flink version are released.{*}{*}{*}
*flink-connector-aws: build on pull request / compile_and_test currently
doesn't support 1.16.2 and 1.17.1 flink version.*
*Add the support to build and test for* \{*}1.16.2 and 1.17.1 flink version on
CI pipeline for flink-connector-aws.{*}{*}{*}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
