On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli <eolive...@gmail.com> wrote: > > Sijie, > Thank you for your feedback > Some additional considerations inline > > Il Lun 19 Lug 2021, 06:47 Sijie Guo <guosi...@gmail.com> ha scritto: > > > I don't think this is a big problem. Because people can recompile the > > function and submit the function. Most of the computing/streaming > > engines ask users to recompile the jobs and resubmit the jobs when it > > upgrades to a new version. > > > Unfortunately this is not easily feasible if the org that is managing the > Pulsar service is different from the org who is developing the Functions. > And especially it is quite impossible to prevent service interruption.
I think the problem also occurs in using streaming engines like Spark and Flink. For example, if you are upgrading Flink from one version to the other version, you have to make a save point in the previous version for all the Flink jobs. Upgrade the Flink cluster and resume jobs in a new version. https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/upgrading/ So it is not unreasonable for asking people to do that when dealing with upgrading a centralized computing engine. > > BTW I believe that there is no way to fix this at this point. > > The best approach here is to document this > > behavior. > > > > I agree that the best thing we can do is to document this requirement. > > Therefore we must ensure in the future that we won't fall again into this > kind of issues. > > Pulsar is becoming more and more used by large enterprises and backward > compatibility is a big value. > > Fortunately not all the Functions need rebuilding. > > > > > > Also, if you are using Kubernetes runtime to schedule functions, you > > are not really impacted. > > > > This is partially correct, because you can wait to upgrade the workers pod, > but there is no fine grained control over which version of each pod will > be running your function, especially in a big cluster with many tenants and > functions with this problem Updating the worker pod will not force all the existing running functions to upgrade. So you can upgrade individual functions after upgrading the worker pods. - Sijie > > > Enrico > > > > - Sijie > > > > On Fri, Jul 16, 2021 at 2:44 AM Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > > Hello, > > > I have reported this issue [1] about upgrading from Pulsar 2.7 to 2.8. > > > More information is on the ticket, but the short version of the story is > > > that > > > in Pulsar 2.8 we introduced a breaking change in the Schema API, by > > > switching SchemaInfo from a class to an interface. > > > > > > This leads to an IncompatibleClassChangeError when you have a Function > > or > > > a Connector that is using Schema.JSON(Pojo.class) and you upgrade your > > > Pulsar cluster (the functions worker pod for instance) from Pulsar 2.7.x > > to > > > Pulsar 2.8.0. > > > > > > The bad problem is that you cannot upgrade Pulsar without interrupting > > the > > > service and coordinating with the upgrade of the Functions. > > > Your functions need to be recompiled against the Pulsar 2.8 API and > > > deployed again in production. > > > > > > I have tried to move back SchemaInfo to an "abstract class" but without > > > success, because then you fall into errors. > > > > > > I am not sure there is a way to provide a good "upgrade path" for > > > Functions/IO users. > > > > > > If we do not find a way we have to document the upgrade in the official > > > Pulsar Documentation. > > > > > > We must do our best to prevent users from falling again into this bad > > > situation. > > > > > > Any suggestions or thoughts ? > > > > > > Regards > > > Enrico > > > > > > [1] https://github.com/apache/pulsar/issues/11338 > >
