Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
pr is created: https://github.com/apache/pulsar/pull/20455 Pengcheng Jiang 于2023年6月1日周四 15:49写道： > Sure, I will create a PR and update the issue when development is done > > Asaf Mesika 于2023年5月31日周三 16:39写道： > >> Pengcheng, would you be willing to be the inaugural PIP in our PIP >> submission process? >> Yesterday, we officially moved from the GitHub issue to a markdown file >> for >> PIP submissions. >> >> For you, it basically means moving your proposal to a markdown file and >> submitting a PR (and deleting the content in the github issue, just >> placing >> a link. Next time no need to open github issue) >> >> The process is described step by step here: >> https://github.com/apache/pulsar/blob/master/pip/README.md >> >> Thanks! >> >> Asaf >> >> >> On Wed, May 31, 2023 at 12:55 AM Neng Lu wrote: >> >> > thanks for the improvements, +1 >> > >> > On Tue, May 30, 2023 at 2:20 AM Pengcheng Jiang >> > wrote: >> > >> > > Hi Mesika: >> > > >> > > Thanks for the suggestions, I updated the pip, and for the rest >> > questions: >> > > >> > > 5. yes, all config goes through arguments instead of a file >> > > 6. it should be a JSON string that can be deserialized to a >> `Map> > > Object>`, updated in pip >> > > 7. it should be `pulsar-admin functions localrun` command, updated in >> pip >> > > 8. the `stateStorageServiceUrl` won't be touched >> > > >> > > Sincerely >> > > Pengcheng Jiang >> > > >> > > Asaf Mesika 于2023年5月29日周一 19:53写道： >> > > >> > > > Hi Pengcheng, >> > > > >> > > > Looks like a solid improvement, definitely helping people using >> their >> > own >> > > > state store. >> > > > >> > > > I have a few comments: >> > > > >> > > > 1. Background knowledge should explain what is a state storage >> > > > 2. Move problem description from Background Knowledge to Motivation. >> > > > >> > > > I'm quoting the template to understand what should be included in >> > > > the Background knowledge section: >> > > > >> > > > >> > > > >> > > > 3. `WorkerConfig` - explain briefly what is Worker and how it >> differs >> > > from >> > > > Broker. Should be in background knowledge section. >> > > > >> > > > 4. Background knowledge should explain briefly what is a runtime and >> > > > runtime factory. >> > > > >> > > > 5. >> > > > >> > > > Add a new cli argument to JavaInstanceStarter and LocalRunner so >> > > > > process runtime can pass state related config to them >> > > > >> > > > >> > > > Today all config goes through arguments and not a file? >> > > > >> > > > 6. `--stateStorageConfig` >> > > > What format is the expected value? >> > > > >> > > > 7. `functions local run` >> > > > What is this? >> > > > >> > > > 8. Are you keeping `stateStorageServiceUrl`? Maybe people rely on >> it? >> > > > >> > > > 9. Don't forget to include link to discussion thread using Apache >> Pony >> > > Mail >> > > > >> > > > >> > > > On Mon, May 29, 2023 at 10:44 AM Rui Fu wrote: >> > > > >> > > > > Hi Pengcheng, >> > > > > >> > > > > Thanks for bringing this up, the PIP lgtm, +1. >> > > > > >> > > > > Best, >> > > > > >> > > > > Rui Fu >> > > > > On May 29, 2023 at 13:52 +0800, Enrico Olivelli < >> eolive...@gmail.com >> > >, >> > > > > wrote: >> > > > > > Looks good >> > > > > > +1 >> > > > > > >> > > > > > Enrico >> > > > > > >> > > > > > Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang >> > > > > > ha scritto: >> > > > > > >> > > > > > > Dear Pulsar community, >> > > > > > > >> > > > > > > I created a pip to make pulsar functions' `StateStoreProvider` >> > > > > configurable >> > > > > > > with custom configurations: >> > > > > https://github.com/apache/pulsar/issues/20419 >> > > > > > > >> > > > > > > Any feedback and suggestions are welcome >> > > > > > > >> > > > > > > Sincerely >> > > > > > > Pengcheng Jiang >> > > > > > > >> > > > > >> > > > >> > > >> > >> >
Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
Sure, I will create a PR and update the issue when development is done Asaf Mesika 于2023年5月31日周三 16:39写道： > Pengcheng, would you be willing to be the inaugural PIP in our PIP > submission process? > Yesterday, we officially moved from the GitHub issue to a markdown file for > PIP submissions. > > For you, it basically means moving your proposal to a markdown file and > submitting a PR (and deleting the content in the github issue, just placing > a link. Next time no need to open github issue) > > The process is described step by step here: > https://github.com/apache/pulsar/blob/master/pip/README.md > > Thanks! > > Asaf > > > On Wed, May 31, 2023 at 12:55 AM Neng Lu wrote: > > > thanks for the improvements, +1 > > > > On Tue, May 30, 2023 at 2:20 AM Pengcheng Jiang > > wrote: > > > > > Hi Mesika: > > > > > > Thanks for the suggestions, I updated the pip, and for the rest > > questions: > > > > > > 5. yes, all config goes through arguments instead of a file > > > 6. it should be a JSON string that can be deserialized to a > `Map > > Object>`, updated in pip > > > 7. it should be `pulsar-admin functions localrun` command, updated in > pip > > > 8. the `stateStorageServiceUrl` won't be touched > > > > > > Sincerely > > > Pengcheng Jiang > > > > > > Asaf Mesika 于2023年5月29日周一 19:53写道： > > > > > > > Hi Pengcheng, > > > > > > > > Looks like a solid improvement, definitely helping people using their > > own > > > > state store. > > > > > > > > I have a few comments: > > > > > > > > 1. Background knowledge should explain what is a state storage > > > > 2. Move problem description from Background Knowledge to Motivation. > > > > > > > > I'm quoting the template to understand what should be included in > > > > the Background knowledge section: > > > > > > > > > > > > > > > > 3. `WorkerConfig` - explain briefly what is Worker and how it differs > > > from > > > > Broker. Should be in background knowledge section. > > > > > > > > 4. Background knowledge should explain briefly what is a runtime and > > > > runtime factory. > > > > > > > > 5. > > > > > > > > Add a new cli argument to JavaInstanceStarter and LocalRunner so > > > > > process runtime can pass state related config to them > > > > > > > > > > > > Today all config goes through arguments and not a file? > > > > > > > > 6. `--stateStorageConfig` > > > > What format is the expected value? > > > > > > > > 7. `functions local run` > > > > What is this? > > > > > > > > 8. Are you keeping `stateStorageServiceUrl`? Maybe people rely on it? > > > > > > > > 9. Don't forget to include link to discussion thread using Apache > Pony > > > Mail > > > > > > > > > > > > On Mon, May 29, 2023 at 10:44 AM Rui Fu wrote: > > > > > > > > > Hi Pengcheng, > > > > > > > > > > Thanks for bringing this up, the PIP lgtm, +1. > > > > > > > > > > Best, > > > > > > > > > > Rui Fu > > > > > On May 29, 2023 at 13:52 +0800, Enrico Olivelli < > eolive...@gmail.com > > >, > > > > > wrote: > > > > > > Looks good > > > > > > +1 > > > > > > > > > > > > Enrico > > > > > > > > > > > > Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang > > > > > > ha scritto: > > > > > > > > > > > > > Dear Pulsar community, > > > > > > > > > > > > > > I created a pip to make pulsar functions' `StateStoreProvider` > > > > > configurable > > > > > > > with custom configurations: > > > > > https://github.com/apache/pulsar/issues/20419 > > > > > > > > > > > > > > Any feedback and suggestions are welcome > > > > > > > > > > > > > > Sincerely > > > > > > > Pengcheng Jiang > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
Pengcheng, would you be willing to be the inaugural PIP in our PIP submission process? Yesterday, we officially moved from the GitHub issue to a markdown file for PIP submissions. For you, it basically means moving your proposal to a markdown file and submitting a PR (and deleting the content in the github issue, just placing a link. Next time no need to open github issue) The process is described step by step here: https://github.com/apache/pulsar/blob/master/pip/README.md Thanks! Asaf On Wed, May 31, 2023 at 12:55 AM Neng Lu wrote: > thanks for the improvements, +1 > > On Tue, May 30, 2023 at 2:20 AM Pengcheng Jiang > wrote: > > > Hi Mesika: > > > > Thanks for the suggestions, I updated the pip, and for the rest > questions: > > > > 5. yes, all config goes through arguments instead of a file > > 6. it should be a JSON string that can be deserialized to a `Map > Object>`, updated in pip > > 7. it should be `pulsar-admin functions localrun` command, updated in pip > > 8. the `stateStorageServiceUrl` won't be touched > > > > Sincerely > > Pengcheng Jiang > > > > Asaf Mesika 于2023年5月29日周一 19:53写道： > > > > > Hi Pengcheng, > > > > > > Looks like a solid improvement, definitely helping people using their > own > > > state store. > > > > > > I have a few comments: > > > > > > 1. Background knowledge should explain what is a state storage > > > 2. Move problem description from Background Knowledge to Motivation. > > > > > > I'm quoting the template to understand what should be included in > > > the Background knowledge section: > > > > > > > > > > > > 3. `WorkerConfig` - explain briefly what is Worker and how it differs > > from > > > Broker. Should be in background knowledge section. > > > > > > 4. Background knowledge should explain briefly what is a runtime and > > > runtime factory. > > > > > > 5. > > > > > > Add a new cli argument to JavaInstanceStarter and LocalRunner so > > > > process runtime can pass state related config to them > > > > > > > > > Today all config goes through arguments and not a file? > > > > > > 6. `--stateStorageConfig` > > > What format is the expected value? > > > > > > 7. `functions local run` > > > What is this? > > > > > > 8. Are you keeping `stateStorageServiceUrl`? Maybe people rely on it? > > > > > > 9. Don't forget to include link to discussion thread using Apache Pony > > Mail > > > > > > > > > On Mon, May 29, 2023 at 10:44 AM Rui Fu wrote: > > > > > > > Hi Pengcheng, > > > > > > > > Thanks for bringing this up, the PIP lgtm, +1. > > > > > > > > Best, > > > > > > > > Rui Fu > > > > On May 29, 2023 at 13:52 +0800, Enrico Olivelli >, > > > > wrote: > > > > > Looks good > > > > > +1 > > > > > > > > > > Enrico > > > > > > > > > > Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang > > > > > ha scritto: > > > > > > > > > > > Dear Pulsar community, > > > > > > > > > > > > I created a pip to make pulsar functions' `StateStoreProvider` > > > > configurable > > > > > > with custom configurations: > > > > https://github.com/apache/pulsar/issues/20419 > > > > > > > > > > > > Any feedback and suggestions are welcome > > > > > > > > > > > > Sincerely > > > > > > Pengcheng Jiang > > > > > > > > > > > > > > > >
Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
thanks for the improvements, +1 On Tue, May 30, 2023 at 2:20 AM Pengcheng Jiang wrote: > Hi Mesika: > > Thanks for the suggestions, I updated the pip, and for the rest questions: > > 5. yes, all config goes through arguments instead of a file > 6. it should be a JSON string that can be deserialized to a `Map Object>`, updated in pip > 7. it should be `pulsar-admin functions localrun` command, updated in pip > 8. the `stateStorageServiceUrl` won't be touched > > Sincerely > Pengcheng Jiang > > Asaf Mesika 于2023年5月29日周一 19:53写道： > > > Hi Pengcheng, > > > > Looks like a solid improvement, definitely helping people using their own > > state store. > > > > I have a few comments: > > > > 1. Background knowledge should explain what is a state storage > > 2. Move problem description from Background Knowledge to Motivation. > > > > I'm quoting the template to understand what should be included in > > the Background knowledge section: > > > > > > > > 3. `WorkerConfig` - explain briefly what is Worker and how it differs > from > > Broker. Should be in background knowledge section. > > > > 4. Background knowledge should explain briefly what is a runtime and > > runtime factory. > > > > 5. > > > > Add a new cli argument to JavaInstanceStarter and LocalRunner so > > > process runtime can pass state related config to them > > > > > > Today all config goes through arguments and not a file? > > > > 6. `--stateStorageConfig` > > What format is the expected value? > > > > 7. `functions local run` > > What is this? > > > > 8. Are you keeping `stateStorageServiceUrl`? Maybe people rely on it? > > > > 9. Don't forget to include link to discussion thread using Apache Pony > Mail > > > > > > On Mon, May 29, 2023 at 10:44 AM Rui Fu wrote: > > > > > Hi Pengcheng, > > > > > > Thanks for bringing this up, the PIP lgtm, +1. > > > > > > Best, > > > > > > Rui Fu > > > On May 29, 2023 at 13:52 +0800, Enrico Olivelli , > > > wrote: > > > > Looks good > > > > +1 > > > > > > > > Enrico > > > > > > > > Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang > > > > ha scritto: > > > > > > > > > Dear Pulsar community, > > > > > > > > > > I created a pip to make pulsar functions' `StateStoreProvider` > > > configurable > > > > > with custom configurations: > > > https://github.com/apache/pulsar/issues/20419 > > > > > > > > > > Any feedback and suggestions are welcome > > > > > > > > > > Sincerely > > > > > Pengcheng Jiang > > > > > > > > > > >
Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
Hi Mesika: Thanks for the suggestions, I updated the pip, and for the rest questions: 5. yes, all config goes through arguments instead of a file 6. it should be a JSON string that can be deserialized to a `Map`, updated in pip 7. it should be `pulsar-admin functions localrun` command, updated in pip 8. the `stateStorageServiceUrl` won't be touched Sincerely Pengcheng Jiang Asaf Mesika 于2023年5月29日周一 19:53写道： > Hi Pengcheng, > > Looks like a solid improvement, definitely helping people using their own > state store. > > I have a few comments: > > 1. Background knowledge should explain what is a state storage > 2. Move problem description from Background Knowledge to Motivation. > > I'm quoting the template to understand what should be included in > the Background knowledge section: > > > > 3. `WorkerConfig` - explain briefly what is Worker and how it differs from > Broker. Should be in background knowledge section. > > 4. Background knowledge should explain briefly what is a runtime and > runtime factory. > > 5. > > Add a new cli argument to JavaInstanceStarter and LocalRunner so > > process runtime can pass state related config to them > > > Today all config goes through arguments and not a file? > > 6. `--stateStorageConfig` > What format is the expected value? > > 7. `functions local run` > What is this? > > 8. Are you keeping `stateStorageServiceUrl`? Maybe people rely on it? > > 9. Don't forget to include link to discussion thread using Apache Pony Mail > > > On Mon, May 29, 2023 at 10:44 AM Rui Fu wrote: > > > Hi Pengcheng, > > > > Thanks for bringing this up, the PIP lgtm, +1. > > > > Best, > > > > Rui Fu > > On May 29, 2023 at 13:52 +0800, Enrico Olivelli , > > wrote: > > > Looks good > > > +1 > > > > > > Enrico > > > > > > Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang > > > ha scritto: > > > > > > > Dear Pulsar community, > > > > > > > > I created a pip to make pulsar functions' `StateStoreProvider` > > configurable > > > > with custom configurations: > > https://github.com/apache/pulsar/issues/20419 > > > > > > > > Any feedback and suggestions are welcome > > > > > > > > Sincerely > > > > Pengcheng Jiang > > > > > > >
Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
Hi Pengcheng, Looks like a solid improvement, definitely helping people using their own state store. I have a few comments: 1. Background knowledge should explain what is a state storage 2. Move problem description from Background Knowledge to Motivation. I'm quoting the template to understand what should be included in the Background knowledge section: 3. `WorkerConfig` - explain briefly what is Worker and how it differs from Broker. Should be in background knowledge section. 4. Background knowledge should explain briefly what is a runtime and runtime factory. 5. Add a new cli argument to JavaInstanceStarter and LocalRunner so > process runtime can pass state related config to them Today all config goes through arguments and not a file? 6. `--stateStorageConfig` What format is the expected value? 7. `functions local run` What is this? 8. Are you keeping `stateStorageServiceUrl`? Maybe people rely on it? 9. Don't forget to include link to discussion thread using Apache Pony Mail On Mon, May 29, 2023 at 10:44 AM Rui Fu wrote: > Hi Pengcheng, > > Thanks for bringing this up, the PIP lgtm, +1. > > Best, > > Rui Fu > On May 29, 2023 at 13:52 +0800, Enrico Olivelli , > wrote: > > Looks good > > +1 > > > > Enrico > > > > Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang > > ha scritto: > > > > > Dear Pulsar community, > > > > > > I created a pip to make pulsar functions' `StateStoreProvider` > configurable > > > with custom configurations: > https://github.com/apache/pulsar/issues/20419 > > > > > > Any feedback and suggestions are welcome > > > > > > Sincerely > > > Pengcheng Jiang > > > >
Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
Hi Pengcheng, Thanks for bringing this up, the PIP lgtm, +1. Best, Rui Fu On May 29, 2023 at 13:52 +0800, Enrico Olivelli , wrote: > Looks good > +1 > > Enrico > > Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang > ha scritto: > > > Dear Pulsar community, > > > > I created a pip to make pulsar functions' `StateStoreProvider` configurable > > with custom configurations: https://github.com/apache/pulsar/issues/20419 > > > > Any feedback and suggestions are welcome > > > > Sincerely > > Pengcheng Jiang > >
Re: [DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
Looks good +1 Enrico Il Lun 29 Mag 2023, 04:47 Pengcheng Jiang ha scritto: > Dear Pulsar community, > > I created a pip to make pulsar functions' `StateStoreProvider` configurable > with custom configurations: https://github.com/apache/pulsar/issues/20419 > > Any feedback and suggestions are welcome > > Sincerely > Pengcheng Jiang >
[DISCUSS] PIP-272 Add a `StateStoreConfig` to the `WorkerConfig`
Dear Pulsar community, I created a pip to make pulsar functions' `StateStoreProvider` configurable with custom configurations: https://github.com/apache/pulsar/issues/20419 Any feedback and suggestions are welcome Sincerely Pengcheng Jiang
