[GitHub] [pulsar-client-node] massakam merged pull request #163: Use Buffer::Copy instead of Buffer::New
massakam merged pull request #163: URL: https://github.com/apache/pulsar-client-node/pull/163 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] massakam commented on pull request #163: Use Buffer::Copy instead of Buffer::New
massakam commented on pull request #163: URL: https://github.com/apache/pulsar-client-node/pull/163#issuecomment-883039565 I got it. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai commented on pull request #163: Use Buffer::Copy instead of Buffer::New
hrsakai commented on pull request #163:
URL:
https://github.com/apache/pulsar-client-node/pull/163#issuecomment-883036429
> The following part doesn't need to be modified because ptr is not
shared_ptr, right?
This issue seems to occure if adding same pointer to `Napi::Buffer::New`,
whether shared_ptr or not.
For example, following code occures the error:
```
Napi::Value MessageId::Serialize(const Napi::CallbackInfo ) {
Napi::Env env = info.Env();
int len;
void *ptr = pulsar_message_id_serialize(GetCMessageId(), );
// OK
Napi::Buffer::New(env, (char *)ptr, len, serializeFinalizeCallback);
// Check failed: result.second.
return Napi::Buffer::New(env, (char *)ptr, len,
serializeFinalizeCallback);
}
```
In fact, `pulsar_message_id_serialize` executes `memcpy` and returns the
copy each time it is called, so same pointer is not added to
`Napi::Buffer::New` in `MessageId::Serialize` and so we don't need to modify
`MessageId::Serialize`.
https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-client-cpp/lib/c/c_MessageId.cc#L46-L53
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-dotpulsar] DrinkStar opened a new issue #82: How to create a new topic by dotpulsar?
DrinkStar opened a new issue #82: URL: https://github.com/apache/pulsar-dotpulsar/issues/82 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] massakam commented on pull request #163: Use Buffer::Copy instead of Buffer::New
massakam commented on pull request #163: URL: https://github.com/apache/pulsar-client-node/pull/163#issuecomment-883013636 The following part doesn't need to be modified because `ptr` is not shared_ptr, right? https://github.com/apache/pulsar-client-node/blob/07c6b1dd8132aed4e3d50ba182c25a7e8e3cf378/src/MessageId.cc#L100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai opened a new pull request #163: Use Buffer::Copy instead of Buffer::New
hrsakai opened a new pull request #163: URL: https://github.com/apache/pulsar-client-node/pull/163 In our enviroment, sometimes we get following error. ``` # # Fatal error in , line 0 # Check failed: result.second. # # # #FailureMessage Object: 0x7ffe7bd5eda0 1: 0xa92851 [node] 2: 0x1a1ac64 V8_Fatal(char const*, ...) [node] 3: 0xe7e119 v8::internal::GlobalBackingStoreRegistry::Register(std::shared_ptr) [node] 4: 0xbc7e48 v8::ArrayBuffer::GetBackingStore() [node] 5: 0x9e3a80 napi_get_typedarray_info [node] 6: 0x7f815dd9f669 Message::GetData(Napi::CallbackInfo const&) [/root/pulsar-client-node/build/Release/Pulsar.node] 7: 0x7f815dda3aa1 Napi::ObjectWrap::InstanceMethodCallbackWrapper(napi_env__*, napi_callback_info__*) [/root/pulsar-client-node/build/Release/Pulsar.node] 8: 0x9da63f [node] 9: 0xc06acb [node] 10: 0xc08076 [node] 11: 0xc086f6 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node] 12: 0x1427159 [node] Illegal instruction (core dumped) ``` ## Modification I use `Napi::Buffer::Copy` instead of `Napi::Buffer::New`. ## Steps to reproduce One easy way to reproduce is to call `msg::getData()` twice. ``` const msg = await consumer.receive(); console.log(msg.getData().toString()); console.log(msg.getData().toString()); ``` This issue occures on node 14 or above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-manager] Huanli-Meng commented on pull request #403: Added chinese version..
Huanli-Meng commented on pull request #403: URL: https://github.com/apache/pulsar-manager/pull/403#issuecomment-882972418 As confirmed with Guowei, he is responsible for updating the doc based on the cloud-manager doc (chinese version) and only review some uncovered contents. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [Doc] Upgrade Docusuraus
Hi all, Just a gentle reminder: If you have any suggestions on the Pulsar website, do not hesitate to leave your comments here (https://docs.google.com/document/d/1IV35SI_F8G8cL-Vuzknc6RTGLK9_edRMpZpnrHvAWNs/edit#) before EOD July 28 (GMT +8). Thanks! On 2021/7/14, 11:48, "Anonymitaet _" wrote: Hi all, Thanks for the discussion. Before upgrading Docusaurus, we've collected some requirements from community and documented here: https://docs.google.com/document/d/1IV35SI_F8G8cL-Vuzknc6RTGLK9_edRMpZpnrHvAWNs/edit#. Feel free to add more if you have any ideas before EOD July 28 (GMT +8). After that, we can take all factors into consideration and make a suitable upgrading plan, thanks! On 2021/7/14, 04:42, "Sijie Guo" wrote: Hi Aaron, Thank you for bringing this up! As part of the docusuras discussion, we have already put together a proposal/plan to revamp the Pulsar website. We are planning to send out the proposal to the community for review and discussion in a couple of days. Would love you to help review the proposal once we send it out. - Sijie On Tue, Jul 13, 2021 at 12:31 PM Aaron Williams wrote: > > Hello Anonymitaet, PMC, and Pulsar Community, > > I love the idea, anything that enables new developers to get started while making life easier for them, our current contributors, and tech writers is a great idea. > > > > I would propose that we take this as an opportunity to go one step farther and overhaul the entire website. > > As we all know, the current one has a number of issues, for example: > > The Home page isn’t visually appealing and doesn’t tell you what Pulsar is. > > Some examples of other Apache product landing pages: > > Kafka: https://kafka.apache.org/ > > RocketMQ: http://rocketmq.apache.org/ > > Cassandra: https://cassandra.apache.org/ > > > > These are similar projects, but their sites are much more engaging than ours and leaves the visitor with a better impression of what those projects do. Plus from other conversations, it is tough to get metrics from the site. > > > > For an example of a technical issue: when you mouse over the tabs they do not open and when you click on the triangles the tab opens but will not close until you click on it again, so you can end up with something like this (if the image isn’t below, go to pulsar.apache.org and just click on the triangles) > > > > > > > > > > > > There are other issues like not having YouTube videos, testimonials from users, etc. (we each can come with a couple more issues). > > > > So following the Apache Community mantra of if you raise an issue, you are volunteering to fix it. With the PMC’s approval, I am volunteering to organize a group to revamp our website and the surrounding collateral. The group will meet to gather specs and create a mock up and report back to the PMC and the larger Pulsar community; take feedback (using fail-fast/ agile methodology) and improve the design. > > > > Thank you for taking the time to read this and I look forward to working with the community to improve our window to the world. > > > > Thanks, > > Aaron Williams > > > > About Me: > > Since this is my first time posting and I thought that I would introduce myself. I am at DataStax and their community manager for streaming, with one of my major goals to help out with the Pulsar Community. I came from the Linux Foundation’s Edge Umbrella project (LFEdge.org) and its 10 projects, where I was the Community Manager/ Developer Advocate, working to grow and strengthen their community. Before that, I was the Global Lead for SAP’s internal makerspace and community space called the d-shop, where we had over 30 locations around the world. Thus working with communities is what I have done for the last 10 years of my career. And I look forward to working with all of you and helping to grow this community. > > > > On Tue, Jul 13, 2021 at 6:52 AM Guangning E wrote: >> >> +1 >> >> Enrico Olivelli 于2021年7月13日周二 下午4:14写道： >>> >>> +1 >>> >>> Enrico >>> >>> Il giorno mar 13 lug 2021 alle ore 10:12 Sijie Guo ha scritto: +1 On Mon, Jul 12, 2021 at 8:25 PM Anonymitaet _ wrote: > > Hi
Re: [DISCUSS] Releasing Pulsar-client-go 0.6.0
+1 Thanks Ming On Mon, 19 Jul 2021 at 18:55, Neng Lu wrote: > +1 > > Neng Lu > > On 2021/07/19 08:44:11 "r...@apache.org" wrote: > > Hello Everyone: > > > > I hope you’ve all been doing well. In the past two months, we have > > fixed a number of bugs related to connection leaks and added > > some new features. For more information refer to: > > > > https://github.com/apache/pulsar-client-go/milestone/7?closed=1 > > > > For that reason, I think we should be releasing a 0.6.0 version with > > what we have today. > > > > -- > > Thanks > > Xiaolong Ran > > > -- Powered by Pulsar Engineering, Kesque
Re: [DISCUSS] Releasing Pulsar-client-go 0.6.0
+1 Neng Lu On 2021/07/19 08:44:11 "r...@apache.org" wrote: > Hello Everyone: > > I hope you’ve all been doing well. In the past two months, we have > fixed a number of bugs related to connection leaks and added > some new features. For more information refer to: > > https://github.com/apache/pulsar-client-go/milestone/7?closed=1 > > For that reason, I think we should be releasing a 0.6.0 version with > what we have today. > > -- > Thanks > Xiaolong Ran >
Re: Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
Based on my local test, it's fine for String Schema. On 2021/07/19 18:47:49 Devin Bost wrote: > > This leads to an IncompatibleClassChangeError when you have a Function or > > a Connector that is using Schema.JSON(Pojo.class) > > I just noticed this detail. Do we have a sense of how often people are > using Schema.JSON in Functions/Connectors? > Most of our functions are using a string schema, so it's not clear to me if > they would be impacted. > > Devin G. Bost > > > On Mon, Jul 19, 2021 at 12:41 PM Devin Bost wrote: > > > > I think Sijie is referring to using KubernetesRuntime to deploy functions > > > where each function/source/sink runs as an independent statefulset in > > K8s. > > > In this scenario, it is possible to have fine grained control over which > > > version of the function container the function is using. > > > > Not everybody is using the KubernetesRuntime yet (especially since the > > Helm charts aren't feature-complete), and it appears that those who aren't > > running KubernetesRuntime would be impacted the most by this issue. > > > > Devin G. Bost > > > > > > On Mon, Jul 19, 2021 at 12:36 PM Devin Bost wrote: > > > >> > For example, if you are upgrading Flink from one version to the other > >> > version, you have to make a save point in the previous version for all > >> > the Flink jobs. > >> > Upgrade the Flink cluster and resume jobs in a new version. > >> > > >> > > >> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/upgrading/ > >> > > >> > So it is not unreasonable for asking people to do that when dealing > >> > with upgrading a centralized computing engine. > >> > >> One difference with Flink is that organizations running Flink in job mode > >> or application mode can upgrade jobs independently of one another, so teams > >> can upgrade jobs when they are ready without impacting other teams. In the > >> Pulsar case, Pulsar is multi-tenant, so upgrading the entire cluster would > >> break every tenant simultaneously and would block the flow of all messages > >> until all functions are upgraded. If one team takes a year to upgrade their > >> one function, the cluster could not be upgraded until that happened. Also, > >> after all the functions have been upgraded, there would be production > >> downtime while deploying all the upgraded functions, which would be a major > >> outage... It might be possible to write a script to speed up the deployment > >> to shrink the outage window, but there's currently a bug that wipes out > >> existing userConfigs when a function is upgraded, so that adds to the > >> complexity of upgrading all the functions since someone would need to know > >> all the userConfigs for all the functions. > >> > >> So, I don't think we're really comparing the same things here. > >> > >> Devin G. Bost > >> > >> > >> On Mon, Jul 19, 2021 at 12:17 PM Sijie Guo wrote: > >> > >>> On Mon, Jul 19, 2021 at 10:32 AM Jerry Peng > >>> wrote: > >>> > > >>> > I agree that the best we can do right now is to just clearly document > >>> this > >>> > as a potential problem when updating 2.7 to 2.8. > >>> > > >>> > We should definitely make every attempt to not make BC breaking > >>> changes. > >>> > However, there are times when we have to make these tough decisions > >>> for one > >>> > reason or another. The bigger problem I see here is not necessarily a > >>> BC > >>> > breaking change occurred, but rather we didn't know about it > >>> beforehand so > >>> > we can clearly document this caveat when 2.8 is released. Perhaps > >>> this is > >>> > where we can improve our backwards compatibility testing. We already > >>> have > >>> > some but probably not enough as highlighted by this case. > >>> > > >>> > In regards to > >>> > > >>> > This is partially correct, because you can wait to upgrade the workers > >>> pod, > >>> > > but there is no fine grained control over which version of each pod > >>> will > >>> > > be running your function, especially in a big cluster with many > >>> tenants and > >>> > > functions with this problem > >>> > > > >>> > > >>> > > >>> > I think Sijie is referring to using KubernetesRuntime to deploy > >>> functions > >>> > where each function/source/sink runs as an independent statefulset in > >>> K8s. > >>> > In this scenario, it is possible to have fine grained control over > >>> which > >>> > version of the function container the function is using. There > >>> currently > >>> > might not be tools to easily allow users to do this but using kubectl > >>> one > >>> > can definitely determine which container version is running and > >>> potentially > >>> > update the container version on a per function basis. > >>> > >>> Jerry - Thank you! That was what I meant. > >>> > >>> > > >>> > Best, > >>> > > >>> > Jerry > >>> > > >>> > On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli > >>> > wrote: > >>> > > >>> > > Sijie, > >>> > > Thank you for your feedback > >>> > > Some additional considerations inline > >>> > > > >>> > >
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
Il Lun 19 Lug 2021, 20:48 Devin Bost ha scritto: > > This leads to an IncompatibleClassChangeError when you have a Function > or > > a Connector that is using Schema.JSON(Pojo.class) > > I just noticed this detail. Do we have a sense of how often people are > using Schema.JSON in Functions/Connectors? > The case I have found is about a Function that is creating a Pulsar Client and creates a Producer and thus it needs to call a static method of Schema (for instance Schema.JSON). Normally Functions do not behave that way. But Connectors are more likely to be willing to use those methods, especially now in 2.9.0 that we are going to give a full PulsarClient Enrico Most of our functions are using a string schema, so it's not clear to me if > they would be impacted. > > Devin G. Bost > > > On Mon, Jul 19, 2021 at 12:41 PM Devin Bost wrote: > > > > I think Sijie is referring to using KubernetesRuntime to deploy > functions > > > where each function/source/sink runs as an independent statefulset in > > K8s. > > > In this scenario, it is possible to have fine grained control over > which > > > version of the function container the function is using. > > > > Not everybody is using the KubernetesRuntime yet (especially since the > > Helm charts aren't feature-complete), and it appears that those who > aren't > > running KubernetesRuntime would be impacted the most by this issue. > > > > Devin G. Bost > > > > > > On Mon, Jul 19, 2021 at 12:36 PM Devin Bost > wrote: > > > >> > For example, if you are upgrading Flink from one version to the other > >> > version, you have to make a save point in the previous version for all > >> > the Flink jobs. > >> > Upgrade the Flink cluster and resume jobs in a new version. > >> > > >> > > >> > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/upgrading/ > >> > > >> > So it is not unreasonable for asking people to do that when dealing > >> > with upgrading a centralized computing engine. > >> > >> One difference with Flink is that organizations running Flink in job > mode > >> or application mode can upgrade jobs independently of one another, so > teams > >> can upgrade jobs when they are ready without impacting other teams. In > the > >> Pulsar case, Pulsar is multi-tenant, so upgrading the entire cluster > would > >> break every tenant simultaneously and would block the flow of all > messages > >> until all functions are upgraded. If one team takes a year to upgrade > their > >> one function, the cluster could not be upgraded until that happened. > Also, > >> after all the functions have been upgraded, there would be production > >> downtime while deploying all the upgraded functions, which would be a > major > >> outage... It might be possible to write a script to speed up the > deployment > >> to shrink the outage window, but there's currently a bug that wipes out > >> existing userConfigs when a function is upgraded, so that adds to the > >> complexity of upgrading all the functions since someone would need to > know > >> all the userConfigs for all the functions. > >> > >> So, I don't think we're really comparing the same things here. > >> > >> Devin G. Bost > >> > >> > >> On Mon, Jul 19, 2021 at 12:17 PM Sijie Guo wrote: > >> > >>> On Mon, Jul 19, 2021 at 10:32 AM Jerry Peng < > jerry.boyang.p...@gmail.com> > >>> wrote: > >>> > > >>> > I agree that the best we can do right now is to just clearly document > >>> this > >>> > as a potential problem when updating 2.7 to 2.8. > >>> > > >>> > We should definitely make every attempt to not make BC breaking > >>> changes. > >>> > However, there are times when we have to make these tough decisions > >>> for one > >>> > reason or another. The bigger problem I see here is not necessarily a > >>> BC > >>> > breaking change occurred, but rather we didn't know about it > >>> beforehand so > >>> > we can clearly document this caveat when 2.8 is released. Perhaps > >>> this is > >>> > where we can improve our backwards compatibility testing. We already > >>> have > >>> > some but probably not enough as highlighted by this case. > >>> > > >>> > In regards to > >>> > > >>> > This is partially correct, because you can wait to upgrade the > workers > >>> pod, > >>> > > but there is no fine grained control over which version of each > pod > >>> will > >>> > > be running your function, especially in a big cluster with many > >>> tenants and > >>> > > functions with this problem > >>> > > > >>> > > >>> > > >>> > I think Sijie is referring to using KubernetesRuntime to deploy > >>> functions > >>> > where each function/source/sink runs as an independent statefulset in > >>> K8s. > >>> > In this scenario, it is possible to have fine grained control over > >>> which > >>> > version of the function container the function is using. There > >>> currently > >>> > might not be tools to easily allow users to do this but using kubectl > >>> one > >>> > can definitely determine which container version is running and > >>>
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
> This leads to an IncompatibleClassChangeError when you have a Function or > a Connector that is using Schema.JSON(Pojo.class) I just noticed this detail. Do we have a sense of how often people are using Schema.JSON in Functions/Connectors? Most of our functions are using a string schema, so it's not clear to me if they would be impacted. Devin G. Bost On Mon, Jul 19, 2021 at 12:41 PM Devin Bost wrote: > > I think Sijie is referring to using KubernetesRuntime to deploy functions > > where each function/source/sink runs as an independent statefulset in > K8s. > > In this scenario, it is possible to have fine grained control over which > > version of the function container the function is using. > > Not everybody is using the KubernetesRuntime yet (especially since the > Helm charts aren't feature-complete), and it appears that those who aren't > running KubernetesRuntime would be impacted the most by this issue. > > Devin G. Bost > > > On Mon, Jul 19, 2021 at 12:36 PM Devin Bost wrote: > >> > For example, if you are upgrading Flink from one version to the other >> > version, you have to make a save point in the previous version for all >> > the Flink jobs. >> > Upgrade the Flink cluster and resume jobs in a new version. >> > >> > >> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/upgrading/ >> > >> > So it is not unreasonable for asking people to do that when dealing >> > with upgrading a centralized computing engine. >> >> One difference with Flink is that organizations running Flink in job mode >> or application mode can upgrade jobs independently of one another, so teams >> can upgrade jobs when they are ready without impacting other teams. In the >> Pulsar case, Pulsar is multi-tenant, so upgrading the entire cluster would >> break every tenant simultaneously and would block the flow of all messages >> until all functions are upgraded. If one team takes a year to upgrade their >> one function, the cluster could not be upgraded until that happened. Also, >> after all the functions have been upgraded, there would be production >> downtime while deploying all the upgraded functions, which would be a major >> outage... It might be possible to write a script to speed up the deployment >> to shrink the outage window, but there's currently a bug that wipes out >> existing userConfigs when a function is upgraded, so that adds to the >> complexity of upgrading all the functions since someone would need to know >> all the userConfigs for all the functions. >> >> So, I don't think we're really comparing the same things here. >> >> Devin G. Bost >> >> >> On Mon, Jul 19, 2021 at 12:17 PM Sijie Guo wrote: >> >>> On Mon, Jul 19, 2021 at 10:32 AM Jerry Peng >>> wrote: >>> > >>> > I agree that the best we can do right now is to just clearly document >>> this >>> > as a potential problem when updating 2.7 to 2.8. >>> > >>> > We should definitely make every attempt to not make BC breaking >>> changes. >>> > However, there are times when we have to make these tough decisions >>> for one >>> > reason or another. The bigger problem I see here is not necessarily a >>> BC >>> > breaking change occurred, but rather we didn't know about it >>> beforehand so >>> > we can clearly document this caveat when 2.8 is released. Perhaps >>> this is >>> > where we can improve our backwards compatibility testing. We already >>> have >>> > some but probably not enough as highlighted by this case. >>> > >>> > In regards to >>> > >>> > This is partially correct, because you can wait to upgrade the workers >>> pod, >>> > > but there is no fine grained control over which version of each pod >>> will >>> > > be running your function, especially in a big cluster with many >>> tenants and >>> > > functions with this problem >>> > > >>> > >>> > >>> > I think Sijie is referring to using KubernetesRuntime to deploy >>> functions >>> > where each function/source/sink runs as an independent statefulset in >>> K8s. >>> > In this scenario, it is possible to have fine grained control over >>> which >>> > version of the function container the function is using. There >>> currently >>> > might not be tools to easily allow users to do this but using kubectl >>> one >>> > can definitely determine which container version is running and >>> potentially >>> > update the container version on a per function basis. >>> >>> Jerry - Thank you! That was what I meant. >>> >>> > >>> > Best, >>> > >>> > Jerry >>> > >>> > On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli >>> > wrote: >>> > >>> > > Sijie, >>> > > Thank you for your feedback >>> > > Some additional considerations inline >>> > > >>> > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: >>> > > >>> > > > I don't think this is a big problem. Because people can recompile >>> the >>> > > > function and submit the function. Most of the computing/streaming >>> > > > engines ask users to recompile the jobs and resubmit the jobs when >>> it >>> > > > upgrades to a new version.
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
> I think Sijie is referring to using KubernetesRuntime to deploy functions > where each function/source/sink runs as an independent statefulset in K8s. > In this scenario, it is possible to have fine grained control over which > version of the function container the function is using. Not everybody is using the KubernetesRuntime yet (especially since the Helm charts aren't feature-complete), and it appears that those who aren't running KubernetesRuntime would be impacted the most by this issue. Devin G. Bost On Mon, Jul 19, 2021 at 12:36 PM Devin Bost wrote: > > For example, if you are upgrading Flink from one version to the other > > version, you have to make a save point in the previous version for all > > the Flink jobs. > > Upgrade the Flink cluster and resume jobs in a new version. > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/upgrading/ > > > > So it is not unreasonable for asking people to do that when dealing > > with upgrading a centralized computing engine. > > One difference with Flink is that organizations running Flink in job mode > or application mode can upgrade jobs independently of one another, so teams > can upgrade jobs when they are ready without impacting other teams. In the > Pulsar case, Pulsar is multi-tenant, so upgrading the entire cluster would > break every tenant simultaneously and would block the flow of all messages > until all functions are upgraded. If one team takes a year to upgrade their > one function, the cluster could not be upgraded until that happened. Also, > after all the functions have been upgraded, there would be production > downtime while deploying all the upgraded functions, which would be a major > outage... It might be possible to write a script to speed up the deployment > to shrink the outage window, but there's currently a bug that wipes out > existing userConfigs when a function is upgraded, so that adds to the > complexity of upgrading all the functions since someone would need to know > all the userConfigs for all the functions. > > So, I don't think we're really comparing the same things here. > > Devin G. Bost > > > On Mon, Jul 19, 2021 at 12:17 PM Sijie Guo wrote: > >> On Mon, Jul 19, 2021 at 10:32 AM Jerry Peng >> wrote: >> > >> > I agree that the best we can do right now is to just clearly document >> this >> > as a potential problem when updating 2.7 to 2.8. >> > >> > We should definitely make every attempt to not make BC breaking changes. >> > However, there are times when we have to make these tough decisions for >> one >> > reason or another. The bigger problem I see here is not necessarily a BC >> > breaking change occurred, but rather we didn't know about it beforehand >> so >> > we can clearly document this caveat when 2.8 is released. Perhaps this >> is >> > where we can improve our backwards compatibility testing. We already >> have >> > some but probably not enough as highlighted by this case. >> > >> > In regards to >> > >> > This is partially correct, because you can wait to upgrade the workers >> pod, >> > > but there is no fine grained control over which version of each pod >> will >> > > be running your function, especially in a big cluster with many >> tenants and >> > > functions with this problem >> > > >> > >> > >> > I think Sijie is referring to using KubernetesRuntime to deploy >> functions >> > where each function/source/sink runs as an independent statefulset in >> K8s. >> > In this scenario, it is possible to have fine grained control over which >> > version of the function container the function is using. There >> currently >> > might not be tools to easily allow users to do this but using kubectl >> one >> > can definitely determine which container version is running and >> potentially >> > update the container version on a per function basis. >> >> Jerry - Thank you! That was what I meant. >> >> > >> > Best, >> > >> > Jerry >> > >> > On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli >> > wrote: >> > >> > > Sijie, >> > > Thank you for your feedback >> > > Some additional considerations inline >> > > >> > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: >> > > >> > > > I don't think this is a big problem. Because people can recompile >> the >> > > > function and submit the function. Most of the computing/streaming >> > > > engines ask users to recompile the jobs and resubmit the jobs when >> it >> > > > upgrades to a new version. >> > > >> > > >> > > Unfortunately this is not easily feasible if the org that is managing >> the >> > > Pulsar service is different from the org who is developing the >> Functions. >> > > And especially it is quite impossible to prevent service interruption. >> > > >> > > BTW I believe that there is no way to fix this at this point. >> > > >> > > The best approach here is to document this >> > > > behavior. >> > > > >> > > >> > > I agree that the best thing we can do is to document this requirement. >> > > >> > > Therefore we must ensure in the
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
> For example, if you are upgrading Flink from one version to the other > version, you have to make a save point in the previous version for all > the Flink jobs. > Upgrade the Flink cluster and resume jobs in a new version. > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/upgrading/ > > So it is not unreasonable for asking people to do that when dealing > with upgrading a centralized computing engine. One difference with Flink is that organizations running Flink in job mode or application mode can upgrade jobs independently of one another, so teams can upgrade jobs when they are ready without impacting other teams. In the Pulsar case, Pulsar is multi-tenant, so upgrading the entire cluster would break every tenant simultaneously and would block the flow of all messages until all functions are upgraded. If one team takes a year to upgrade their one function, the cluster could not be upgraded until that happened. Also, after all the functions have been upgraded, there would be production downtime while deploying all the upgraded functions, which would be a major outage... It might be possible to write a script to speed up the deployment to shrink the outage window, but there's currently a bug that wipes out existing userConfigs when a function is upgraded, so that adds to the complexity of upgrading all the functions since someone would need to know all the userConfigs for all the functions. So, I don't think we're really comparing the same things here. Devin G. Bost On Mon, Jul 19, 2021 at 12:17 PM Sijie Guo wrote: > On Mon, Jul 19, 2021 at 10:32 AM Jerry Peng > wrote: > > > > I agree that the best we can do right now is to just clearly document > this > > as a potential problem when updating 2.7 to 2.8. > > > > We should definitely make every attempt to not make BC breaking changes. > > However, there are times when we have to make these tough decisions for > one > > reason or another. The bigger problem I see here is not necessarily a BC > > breaking change occurred, but rather we didn't know about it beforehand > so > > we can clearly document this caveat when 2.8 is released. Perhaps this > is > > where we can improve our backwards compatibility testing. We already > have > > some but probably not enough as highlighted by this case. > > > > In regards to > > > > This is partially correct, because you can wait to upgrade the workers > pod, > > > but there is no fine grained control over which version of each pod > will > > > be running your function, especially in a big cluster with many > tenants and > > > functions with this problem > > > > > > > > > I think Sijie is referring to using KubernetesRuntime to deploy functions > > where each function/source/sink runs as an independent statefulset in > K8s. > > In this scenario, it is possible to have fine grained control over which > > version of the function container the function is using. There currently > > might not be tools to easily allow users to do this but using kubectl one > > can definitely determine which container version is running and > potentially > > update the container version on a per function basis. > > Jerry - Thank you! That was what I meant. > > > > > Best, > > > > Jerry > > > > On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli > > wrote: > > > > > Sijie, > > > Thank you for your feedback > > > Some additional considerations inline > > > > > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > > > > > > > I don't think this is a big problem. Because people can recompile the > > > > function and submit the function. Most of the computing/streaming > > > > engines ask users to recompile the jobs and resubmit the jobs when it > > > > upgrades to a new version. > > > > > > > > > Unfortunately this is not easily feasible if the org that is managing > the > > > Pulsar service is different from the org who is developing the > Functions. > > > And especially it is quite impossible to prevent service interruption. > > > > > > BTW I believe that there is no way to fix this at this point. > > > > > > The best approach here is to document this > > > > behavior. > > > > > > > > > > I agree that the best thing we can do is to document this requirement. > > > > > > Therefore we must ensure in the future that we won't fall again into > this > > > kind of issues. > > > > > > Pulsar is becoming more and more used by large enterprises and backward > > > compatibility is a big value. > > > > > > Fortunately not all the Functions need rebuilding. > > > > > > > > > > > > > > > > Also, if you are using Kubernetes runtime to schedule functions, you > > > > are not really impacted. > > > > > > > > > > This is partially correct, because you can wait to upgrade the workers > pod, > > > but there is no fine grained control over which version of each pod > will > > > be running your function, especially in a big cluster with many > tenants and > > > functions with this problem > > > > > > > > > Enrico > > > > > > > > > >
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
On Mon, Jul 19, 2021 at 10:32 AM Jerry Peng wrote: > > I agree that the best we can do right now is to just clearly document this > as a potential problem when updating 2.7 to 2.8. > > We should definitely make every attempt to not make BC breaking changes. > However, there are times when we have to make these tough decisions for one > reason or another. The bigger problem I see here is not necessarily a BC > breaking change occurred, but rather we didn't know about it beforehand so > we can clearly document this caveat when 2.8 is released. Perhaps this is > where we can improve our backwards compatibility testing. We already have > some but probably not enough as highlighted by this case. > > In regards to > > This is partially correct, because you can wait to upgrade the workers pod, > > but there is no fine grained control over which version of each pod will > > be running your function, especially in a big cluster with many tenants and > > functions with this problem > > > > > I think Sijie is referring to using KubernetesRuntime to deploy functions > where each function/source/sink runs as an independent statefulset in K8s. > In this scenario, it is possible to have fine grained control over which > version of the function container the function is using. There currently > might not be tools to easily allow users to do this but using kubectl one > can definitely determine which container version is running and potentially > update the container version on a per function basis. Jerry - Thank you! That was what I meant. > > Best, > > Jerry > > On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli > wrote: > > > Sijie, > > Thank you for your feedback > > Some additional considerations inline > > > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > > > > > I don't think this is a big problem. Because people can recompile the > > > function and submit the function. Most of the computing/streaming > > > engines ask users to recompile the jobs and resubmit the jobs when it > > > upgrades to a new version. > > > > > > Unfortunately this is not easily feasible if the org that is managing the > > Pulsar service is different from the org who is developing the Functions. > > And especially it is quite impossible to prevent service interruption. > > > > BTW I believe that there is no way to fix this at this point. > > > > The best approach here is to document this > > > behavior. > > > > > > > I agree that the best thing we can do is to document this requirement. > > > > Therefore we must ensure in the future that we won't fall again into this > > kind of issues. > > > > Pulsar is becoming more and more used by large enterprises and backward > > compatibility is a big value. > > > > Fortunately not all the Functions need rebuilding. > > > > > > > > > > > Also, if you are using Kubernetes runtime to schedule functions, you > > > are not really impacted. > > > > > > > This is partially correct, because you can wait to upgrade the workers pod, > > but there is no fine grained control over which version of each pod will > > be running your function, especially in a big cluster with many tenants and > > functions with this problem > > > > > > Enrico > > > > > > > - Sijie > > > > > > On Fri, Jul 16, 2021 at 2:44 AM Enrico Olivelli > > > wrote: > > > > > > > > Hello, > > > > I have reported this issue [1] about upgrading from Pulsar 2.7 to 2.8. > > > > More information is on the ticket, but the short version of the story > > is > > > > that > > > > in Pulsar 2.8 we introduced a breaking change in the Schema API, by > > > > switching SchemaInfo from a class to an interface. > > > > > > > > This leads to an IncompatibleClassChangeError when you have a Function > > > or > > > > a Connector that is using Schema.JSON(Pojo.class) and you upgrade your > > > > Pulsar cluster (the functions worker pod for instance) from Pulsar > > 2.7.x > > > to > > > > Pulsar 2.8.0. > > > > > > > > The bad problem is that you cannot upgrade Pulsar without interrupting > > > the > > > > service and coordinating with the upgrade of the Functions. > > > > Your functions need to be recompiled against the Pulsar 2.8 API and > > > > deployed again in production. > > > > > > > > I have tried to move back SchemaInfo to an "abstract class" but without > > > > success, because then you fall into errors. > > > > > > > > I am not sure there is a way to provide a good "upgrade path" for > > > > Functions/IO users. > > > > > > > > If we do not find a way we have to document the upgrade in the official > > > > Pulsar Documentation. > > > > > > > > We must do our best to prevent users from falling again into this bad > > > > situation. > > > > > > > > Any suggestions or thoughts ? > > > > > > > > Regards > > > > Enrico > > > > > > > > [1] https://github.com/apache/pulsar/issues/11338 > > > > >
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli wrote: > > Sijie, > Thank you for your feedback > Some additional considerations inline > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > > > I don't think this is a big problem. Because people can recompile the > > function and submit the function. Most of the computing/streaming > > engines ask users to recompile the jobs and resubmit the jobs when it > > upgrades to a new version. > > > Unfortunately this is not easily feasible if the org that is managing the > Pulsar service is different from the org who is developing the Functions. > And especially it is quite impossible to prevent service interruption. I think the problem also occurs in using streaming engines like Spark and Flink. For example, if you are upgrading Flink from one version to the other version, you have to make a save point in the previous version for all the Flink jobs. Upgrade the Flink cluster and resume jobs in a new version. https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/upgrading/ So it is not unreasonable for asking people to do that when dealing with upgrading a centralized computing engine. > > BTW I believe that there is no way to fix this at this point. > > The best approach here is to document this > > behavior. > > > > I agree that the best thing we can do is to document this requirement. > > Therefore we must ensure in the future that we won't fall again into this > kind of issues. > > Pulsar is becoming more and more used by large enterprises and backward > compatibility is a big value. > > Fortunately not all the Functions need rebuilding. > > > > > > Also, if you are using Kubernetes runtime to schedule functions, you > > are not really impacted. > > > > This is partially correct, because you can wait to upgrade the workers pod, > but there is no fine grained control over which version of each pod will > be running your function, especially in a big cluster with many tenants and > functions with this problem Updating the worker pod will not force all the existing running functions to upgrade. So you can upgrade individual functions after upgrading the worker pods. - Sijie > > > Enrico > > > > - Sijie > > > > On Fri, Jul 16, 2021 at 2:44 AM Enrico Olivelli > > wrote: > > > > > > Hello, > > > I have reported this issue [1] about upgrading from Pulsar 2.7 to 2.8. > > > More information is on the ticket, but the short version of the story is > > > that > > > in Pulsar 2.8 we introduced a breaking change in the Schema API, by > > > switching SchemaInfo from a class to an interface. > > > > > > This leads to an IncompatibleClassChangeError when you have a Function > > or > > > a Connector that is using Schema.JSON(Pojo.class) and you upgrade your > > > Pulsar cluster (the functions worker pod for instance) from Pulsar 2.7.x > > to > > > Pulsar 2.8.0. > > > > > > The bad problem is that you cannot upgrade Pulsar without interrupting > > the > > > service and coordinating with the upgrade of the Functions. > > > Your functions need to be recompiled against the Pulsar 2.8 API and > > > deployed again in production. > > > > > > I have tried to move back SchemaInfo to an "abstract class" but without > > > success, because then you fall into errors. > > > > > > I am not sure there is a way to provide a good "upgrade path" for > > > Functions/IO users. > > > > > > If we do not find a way we have to document the upgrade in the official > > > Pulsar Documentation. > > > > > > We must do our best to prevent users from falling again into this bad > > > situation. > > > > > > Any suggestions or thoughts ? > > > > > > Regards > > > Enrico > > > > > > [1] https://github.com/apache/pulsar/issues/11338 > >
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
Is the client at least backwards compatible so that we could have all of our teams upgrade functions before a broker upgrade is scheduled? We have hundreds of functions running in production that are maintained by many different teams and departments. If upgrades of the functions need be all happen while the cluster is unable to process messages (until the upgrade is complete), that would be a blocker for a lot of organizations and could be incompatible with a lot of CI/CD processes. -- Devin G. Bost On Mon, Jul 19, 2021, 11:32 AM Jerry Peng wrote: > I agree that the best we can do right now is to just clearly document this > as a potential problem when updating 2.7 to 2.8. > > We should definitely make every attempt to not make BC breaking changes. > However, there are times when we have to make these tough decisions for one > reason or another. The bigger problem I see here is not necessarily a BC > breaking change occurred, but rather we didn't know about it beforehand so > we can clearly document this caveat when 2.8 is released. Perhaps this is > where we can improve our backwards compatibility testing. We already have > some but probably not enough as highlighted by this case. > > In regards to > > This is partially correct, because you can wait to upgrade the workers pod, > > but there is no fine grained control over which version of each pod will > > be running your function, especially in a big cluster with many tenants > and > > functions with this problem > > > > > I think Sijie is referring to using KubernetesRuntime to deploy functions > where each function/source/sink runs as an independent statefulset in K8s. > In this scenario, it is possible to have fine grained control over which > version of the function container the function is using. There currently > might not be tools to easily allow users to do this but using kubectl one > can definitely determine which container version is running and potentially > update the container version on a per function basis. > > Best, > > Jerry > > On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli > wrote: > > > Sijie, > > Thank you for your feedback > > Some additional considerations inline > > > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > > > > > I don't think this is a big problem. Because people can recompile the > > > function and submit the function. Most of the computing/streaming > > > engines ask users to recompile the jobs and resubmit the jobs when it > > > upgrades to a new version. > > > > > > Unfortunately this is not easily feasible if the org that is managing the > > Pulsar service is different from the org who is developing the Functions. > > And especially it is quite impossible to prevent service interruption. > > > > BTW I believe that there is no way to fix this at this point. > > > > The best approach here is to document this > > > behavior. > > > > > > > I agree that the best thing we can do is to document this requirement. > > > > Therefore we must ensure in the future that we won't fall again into this > > kind of issues. > > > > Pulsar is becoming more and more used by large enterprises and backward > > compatibility is a big value. > > > > Fortunately not all the Functions need rebuilding. > > > > > > > > > > > Also, if you are using Kubernetes runtime to schedule functions, you > > > are not really impacted. > > > > > > > This is partially correct, because you can wait to upgrade the workers > pod, > > but there is no fine grained control over which version of each pod will > > be running your function, especially in a big cluster with many tenants > and > > functions with this problem > > > > > > Enrico > > > > > > > - Sijie > > > > > > On Fri, Jul 16, 2021 at 2:44 AM Enrico Olivelli > > > wrote: > > > > > > > > Hello, > > > > I have reported this issue [1] about upgrading from Pulsar 2.7 to > 2.8. > > > > More information is on the ticket, but the short version of the story > > is > > > > that > > > > in Pulsar 2.8 we introduced a breaking change in the Schema API, by > > > > switching SchemaInfo from a class to an interface. > > > > > > > > This leads to an IncompatibleClassChangeError when you have a > Function > > > or > > > > a Connector that is using Schema.JSON(Pojo.class) and you upgrade > your > > > > Pulsar cluster (the functions worker pod for instance) from Pulsar > > 2.7.x > > > to > > > > Pulsar 2.8.0. > > > > > > > > The bad problem is that you cannot upgrade Pulsar without > interrupting > > > the > > > > service and coordinating with the upgrade of the Functions. > > > > Your functions need to be recompiled against the Pulsar 2.8 API and > > > > deployed again in production. > > > > > > > > I have tried to move back SchemaInfo to an "abstract class" but > without > > > > success, because then you fall into errors. > > > > > > > > I am not sure there is a way to provide a good "upgrade path" for > > > > Functions/IO users. > > > > > > > > If we do not find a way we have to document the
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
I agree that the best we can do right now is to just clearly document this as a potential problem when updating 2.7 to 2.8. We should definitely make every attempt to not make BC breaking changes. However, there are times when we have to make these tough decisions for one reason or another. The bigger problem I see here is not necessarily a BC breaking change occurred, but rather we didn't know about it beforehand so we can clearly document this caveat when 2.8 is released. Perhaps this is where we can improve our backwards compatibility testing. We already have some but probably not enough as highlighted by this case. In regards to This is partially correct, because you can wait to upgrade the workers pod, > but there is no fine grained control over which version of each pod will > be running your function, especially in a big cluster with many tenants and > functions with this problem > I think Sijie is referring to using KubernetesRuntime to deploy functions where each function/source/sink runs as an independent statefulset in K8s. In this scenario, it is possible to have fine grained control over which version of the function container the function is using. There currently might not be tools to easily allow users to do this but using kubectl one can definitely determine which container version is running and potentially update the container version on a per function basis. Best, Jerry On Mon, Jul 19, 2021 at 12:50 AM Enrico Olivelli wrote: > Sijie, > Thank you for your feedback > Some additional considerations inline > > Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > > > I don't think this is a big problem. Because people can recompile the > > function and submit the function. Most of the computing/streaming > > engines ask users to recompile the jobs and resubmit the jobs when it > > upgrades to a new version. > > > Unfortunately this is not easily feasible if the org that is managing the > Pulsar service is different from the org who is developing the Functions. > And especially it is quite impossible to prevent service interruption. > > BTW I believe that there is no way to fix this at this point. > > The best approach here is to document this > > behavior. > > > > I agree that the best thing we can do is to document this requirement. > > Therefore we must ensure in the future that we won't fall again into this > kind of issues. > > Pulsar is becoming more and more used by large enterprises and backward > compatibility is a big value. > > Fortunately not all the Functions need rebuilding. > > > > > > Also, if you are using Kubernetes runtime to schedule functions, you > > are not really impacted. > > > > This is partially correct, because you can wait to upgrade the workers pod, > but there is no fine grained control over which version of each pod will > be running your function, especially in a big cluster with many tenants and > functions with this problem > > > Enrico > > > > - Sijie > > > > On Fri, Jul 16, 2021 at 2:44 AM Enrico Olivelli > > wrote: > > > > > > Hello, > > > I have reported this issue [1] about upgrading from Pulsar 2.7 to 2.8. > > > More information is on the ticket, but the short version of the story > is > > > that > > > in Pulsar 2.8 we introduced a breaking change in the Schema API, by > > > switching SchemaInfo from a class to an interface. > > > > > > This leads to an IncompatibleClassChangeError when you have a Function > > or > > > a Connector that is using Schema.JSON(Pojo.class) and you upgrade your > > > Pulsar cluster (the functions worker pod for instance) from Pulsar > 2.7.x > > to > > > Pulsar 2.8.0. > > > > > > The bad problem is that you cannot upgrade Pulsar without interrupting > > the > > > service and coordinating with the upgrade of the Functions. > > > Your functions need to be recompiled against the Pulsar 2.8 API and > > > deployed again in production. > > > > > > I have tried to move back SchemaInfo to an "abstract class" but without > > > success, because then you fall into errors. > > > > > > I am not sure there is a way to provide a good "upgrade path" for > > > Functions/IO users. > > > > > > If we do not find a way we have to document the upgrade in the official > > > Pulsar Documentation. > > > > > > We must do our best to prevent users from falling again into this bad > > > situation. > > > > > > Any suggestions or thoughts ? > > > > > > Regards > > > Enrico > > > > > > [1] https://github.com/apache/pulsar/issues/11338 > > >
Re: [DISCUSS] Releasing Pulsar-client-go 0.6.0
+1, Thanks Rui Fu 在 2021年7月19日 +0800 PM4:44，r...@apache.org ，写道： > Hello Everyone: > > I hope you’ve all been doing well. In the past two months, we have > fixed a number of bugs related to connection leaks and added > some new features. For more information refer to: > > https://github.com/apache/pulsar-client-go/milestone/7?closed=1 > > For that reason, I think we should be releasing a 0.6.0 version with > what we have today. > > -- > Thanks > Xiaolong Ran
Re: [DISCUSS] Releasing Pulsar-client-go 0.6.0
+1 Enrico Il giorno lun 19 lug 2021 alle ore 10:44 r...@apache.org < ranxiaolong...@gmail.com> ha scritto: > Hello Everyone: > > I hope you’ve all been doing well. In the past two months, we have > fixed a number of bugs related to connection leaks and added > some new features. For more information refer to: > > https://github.com/apache/pulsar-client-go/milestone/7?closed=1 > > For that reason, I think we should be releasing a 0.6.0 version with > what we have today. > > -- > Thanks > Xiaolong Ran >
[DISCUSS] Releasing Pulsar-client-go 0.6.0
Hello Everyone: I hope you’ve all been doing well. In the past two months, we have fixed a number of bugs related to connection leaks and added some new features. For more information refer to: https://github.com/apache/pulsar-client-go/milestone/7?closed=1 For that reason, I think we should be releasing a 0.6.0 version with what we have today. -- Thanks Xiaolong Ran
Re: Problems with Functions/IO in Upgrading Pulsar from 2.7 to 2.8
Sijie, Thank you for your feedback Some additional considerations inline Il Lun 19 Lug 2021, 06:47 Sijie Guo ha scritto: > I don't think this is a big problem. Because people can recompile the > function and submit the function. Most of the computing/streaming > engines ask users to recompile the jobs and resubmit the jobs when it > upgrades to a new version. Unfortunately this is not easily feasible if the org that is managing the Pulsar service is different from the org who is developing the Functions. And especially it is quite impossible to prevent service interruption. BTW I believe that there is no way to fix this at this point. The best approach here is to document this > behavior. > I agree that the best thing we can do is to document this requirement. Therefore we must ensure in the future that we won't fall again into this kind of issues. Pulsar is becoming more and more used by large enterprises and backward compatibility is a big value. Fortunately not all the Functions need rebuilding. > Also, if you are using Kubernetes runtime to schedule functions, you > are not really impacted. > This is partially correct, because you can wait to upgrade the workers pod, but there is no fine grained control over which version of each pod will be running your function, especially in a big cluster with many tenants and functions with this problem Enrico > - Sijie > > On Fri, Jul 16, 2021 at 2:44 AM Enrico Olivelli > wrote: > > > > Hello, > > I have reported this issue [1] about upgrading from Pulsar 2.7 to 2.8. > > More information is on the ticket, but the short version of the story is > > that > > in Pulsar 2.8 we introduced a breaking change in the Schema API, by > > switching SchemaInfo from a class to an interface. > > > > This leads to an IncompatibleClassChangeError when you have a Function > or > > a Connector that is using Schema.JSON(Pojo.class) and you upgrade your > > Pulsar cluster (the functions worker pod for instance) from Pulsar 2.7.x > to > > Pulsar 2.8.0. > > > > The bad problem is that you cannot upgrade Pulsar without interrupting > the > > service and coordinating with the upgrade of the Functions. > > Your functions need to be recompiled against the Pulsar 2.8 API and > > deployed again in production. > > > > I have tried to move back SchemaInfo to an "abstract class" but without > > success, because then you fall into errors. > > > > I am not sure there is a way to provide a good "upgrade path" for > > Functions/IO users. > > > > If we do not find a way we have to document the upgrade in the official > > Pulsar Documentation. > > > > We must do our best to prevent users from falling again into this bad > > situation. > > > > Any suggestions or thoughts ? > > > > Regards > > Enrico > > > > [1] https://github.com/apache/pulsar/issues/11338 >
