Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
-- Matteo Merli On Sun, Dec 19, 2021 at 9:42 PM Lan Liang wrote: > > My two cents +1 , In my case, i sometimes need to run pulsar standalone in > some server with limited resources. > The other point i can think of is maybe schema storage also need to change > from using BK to using local filesystem or some else. Since we're anyway going to have an embedded bookie, I think we're fine to continue using BK for schema storage.
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 On Mon, 20 Dec 2021 at 13:42, Lan Liang wrote: > My two cents +1 , In my case, i sometimes need to run pulsar standalone in > some server with limited resources. > > > > Proposed changes > > The other point i can think of is maybe schema storage also need to change > from using BK to using local filesystem or some else. > > > > > > > Best Regards, > Lan Liang > On 12/17/2021 12:35，Sijie Guo wrote： > +1 > > On Tue, Dec 14, 2021 at 9:18 AM Matteo Merli wrote: > > https://github.com/apache/pulsar/issues/13302 > > Copying here for quoting convenience > > > > > > ## Motivation > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, > where > all the components are started within the context of a single JVM process. > > Users are using the standalone as a way to get quickly started with Pulsar > or > in all the cases where it makes sense to have a single node deployment. > > Right now, the standalone is starting by default with many components, > several of > which are quite complex, since they are designed to be deployed in a > distributed > fashion. > > ## Goal > > Simplify the components of Pulsar standalone to achieve: > > 1. Reduce complexity > 2. Reduce startup time > 3. Reduce memory and CPU footprint of running standalone > > ## Proposed changes > > The proposal here is to change some of the default implementations that are > used for the Pulsar standalone. > > 1. **Metadata Store implementation** --> > Change from ZooKeeper to RocksDB > > 2. **Pulsar functions package backend** --> > Change from using DistributedLog to using local filesystem, storing > the > jars directly in the data folder instead of uploading them into BK. > > 3. **Pulsar functions state store implementation** --> > Change the state store to be backed by a MetadataStore based backed, > with the RocksDB implementation. > > 4. **Table Service** --> > Do not start BK table service by default > > ## Compatibility considerations > > In order to avoid compatibility issues where users have existing Pulsar > standalone services that they want to upgrade without conflicts, we will > follow the principle of keeping the old defaults where there is existing > data on the disk. > > We will add a file, serving the purpose as a flag, in the `data/standalone` > directory, for example `new-2.10-defaults`. > > If the file is present, or if the data directory is completely missing, we > will adopt the new set of default configuration settings. > > If the file is not there, we will continue to use existing defaults and we > will > not break the upgrade operation. > > > > > > -- > Matteo Merli > > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
My two cents +1 , In my case, i sometimes need to run pulsar standalone in some server with limited resources. > Proposed changes The other point i can think of is maybe schema storage also need to change from using BK to using local filesystem or some else. Best Regards, Lan Liang On 12/17/2021 12:35，Sijie Guo wrote： +1 On Tue, Dec 14, 2021 at 9:18 AM Matteo Merli wrote: https://github.com/apache/pulsar/issues/13302 Copying here for quoting convenience ## Motivation Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, where all the components are started within the context of a single JVM process. Users are using the standalone as a way to get quickly started with Pulsar or in all the cases where it makes sense to have a single node deployment. Right now, the standalone is starting by default with many components, several of which are quite complex, since they are designed to be deployed in a distributed fashion. ## Goal Simplify the components of Pulsar standalone to achieve: 1. Reduce complexity 2. Reduce startup time 3. Reduce memory and CPU footprint of running standalone ## Proposed changes The proposal here is to change some of the default implementations that are used for the Pulsar standalone. 1. **Metadata Store implementation** --> Change from ZooKeeper to RocksDB 2. **Pulsar functions package backend** --> Change from using DistributedLog to using local filesystem, storing the jars directly in the data folder instead of uploading them into BK. 3. **Pulsar functions state store implementation** --> Change the state store to be backed by a MetadataStore based backed, with the RocksDB implementation. 4. **Table Service** --> Do not start BK table service by default ## Compatibility considerations In order to avoid compatibility issues where users have existing Pulsar standalone services that they want to upgrade without conflicts, we will follow the principle of keeping the old defaults where there is existing data on the disk. We will add a file, serving the purpose as a flag, in the `data/standalone` directory, for example `new-2.10-defaults`. If the file is present, or if the data directory is completely missing, we will adopt the new set of default configuration settings. If the file is not there, we will continue to use existing defaults and we will not break the upgrade operation. -- Matteo Merli
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 On Tue, Dec 14, 2021 at 9:18 AM Matteo Merli wrote: > https://github.com/apache/pulsar/issues/13302 > > Copying here for quoting convenience > > > > > > ## Motivation > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, > where > all the components are started within the context of a single JVM process. > > Users are using the standalone as a way to get quickly started with Pulsar > or > in all the cases where it makes sense to have a single node deployment. > > Right now, the standalone is starting by default with many components, > several of > which are quite complex, since they are designed to be deployed in a > distributed > fashion. > > ## Goal > > Simplify the components of Pulsar standalone to achieve: > > 1. Reduce complexity > 2. Reduce startup time > 3. Reduce memory and CPU footprint of running standalone > > ## Proposed changes > > The proposal here is to change some of the default implementations that are > used for the Pulsar standalone. > > 1. **Metadata Store implementation** --> > Change from ZooKeeper to RocksDB > > 2. **Pulsar functions package backend** --> > Change from using DistributedLog to using local filesystem, storing > the > jars directly in the data folder instead of uploading them into BK. > > 3. **Pulsar functions state store implementation** --> > Change the state store to be backed by a MetadataStore based backed, > with the RocksDB implementation. > > 4. **Table Service** --> > Do not start BK table service by default > > ## Compatibility considerations > > In order to avoid compatibility issues where users have existing Pulsar > standalone services that they want to upgrade without conflicts, we will > follow the principle of keeping the old defaults where there is existing > data on the disk. > > We will add a file, serving the purpose as a flag, in the `data/standalone` > directory, for example `new-2.10-defaults`. > > If the file is present, or if the data directory is completely missing, we > will adopt the new set of default configuration settings. > > If the file is not there, we will continue to use existing defaults and we > will > not break the upgrade operation. > > > > > > -- > Matteo Merli > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
Hi Enrico, This migration tool is a better-to-have and mostly about metadata store part. Some of our user use Standalone in a long running test environment, with this migration tool they do not need to recreate all the topics and subscriptions manually and take these benefits with new default settings. And furthermore, since we have different kinds of metadata store implementations, metadata migration is a potential requirement. Just some brief thoughts yet. I think we can start a new DISCUSSION on this. Thanks, Haiting On 2021/12/15 11:09:22 Enrico Olivelli wrote: > +1 > > @Haiting > Pulsar Standalone is usually used for local development, I am not sure you > need a migration tool. > You can still keep the configuration to use previously (standalone.conf) > > Enrico > > Il giorno mer 15 dic 2021 alle ore 10:56 Haiting Jiang < > jianghait...@apache.org> ha scritto: > > > +1 (non-binding) > > > > Out of the scope of this PIP, a follow up data migration tool could be > > added to help migration from old settings without data loss. > > > > --- > > Haiting > > > > On 2021/12/15 01:04:21 Hang Chen wrote: > > > +1 > > > > > > Thanks, > > > Hang > > > > > > PengHui Li 于2021年12月15日周三 07:52写道： > > > > > > > > +1 > > > > > > > > Penghui > > > > > > > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli > > wrote: > > > > > > > > > https://github.com/apache/pulsar/issues/13302 > > > > > > > > > > Copying here for quoting convenience > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar > > cluster, > > > > > where > > > > > all the components are started within the context of a single JVM > > process. > > > > > > > > > > Users are using the standalone as a way to get quickly started with > > Pulsar > > > > > or > > > > > in all the cases where it makes sense to have a single node > > deployment. > > > > > > > > > > Right now, the standalone is starting by default with many > > components, > > > > > several of > > > > > which are quite complex, since they are designed to be deployed in a > > > > > distributed > > > > > fashion. > > > > > > > > > > ## Goal > > > > > > > > > > Simplify the components of Pulsar standalone to achieve: > > > > > > > > > > 1. Reduce complexity > > > > > 2. Reduce startup time > > > > > 3. Reduce memory and CPU footprint of running standalone > > > > > > > > > > ## Proposed changes > > > > > > > > > > The proposal here is to change some of the default implementations > > that are > > > > > used for the Pulsar standalone. > > > > > > > > > > 1. **Metadata Store implementation** --> > > > > > Change from ZooKeeper to RocksDB > > > > > > > > > > 2. **Pulsar functions package backend** --> > > > > > Change from using DistributedLog to using local filesystem, > > storing > > > > > the > > > > > jars directly in the data folder instead of uploading them > > into BK. > > > > > > > > > > 3. **Pulsar functions state store implementation** --> > > > > > Change the state store to be backed by a MetadataStore based > > backed, > > > > > with the RocksDB implementation. > > > > > > > > > > 4. **Table Service** --> > > > > > Do not start BK table service by default > > > > > > > > > > ## Compatibility considerations > > > > > > > > > > In order to avoid compatibility issues where users have existing > > Pulsar > > > > > standalone services that they want to upgrade without conflicts, we > > will > > > > > follow the principle of keeping the old defaults where there is > > existing > > > > > data on the disk. > > > > > > > > > > We will add a file, serving the purpose as a flag, in the > > `data/standalone` > > > > > directory, for example `new-2.10-defaults`. > > > > > > > > > > If the file is present, or if the data directory is completely > > missing, we > > > > > will adopt the new set of default configuration settings. > > > > > > > > > > If the file is not there, we will continue to use existing defaults > > and we > > > > > will > > > > > not break the upgrade operation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Matteo Merli > > > > > > > > > > > > > > > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 I think this is a good idea. The tradeoff here is that testing against a standalone "cluster" is even more different than testing against a distributed Pulsar cluster. I think this is fine, but if we accept this PIP, I think we should update our guidance on the Wiki, which currently recommends running release candidate validation tests against a standalone cluster [0]. - Michael [0] https://github.com/apache/pulsar/wiki/Release-Candidate-Validation#validate-pubsub-and-java-functions On Wed, Dec 15, 2021 at 5:09 AM Enrico Olivelli wrote: > > +1 > > @Haiting > Pulsar Standalone is usually used for local development, I am not sure you > need a migration tool. > You can still keep the configuration to use previously (standalone.conf) > > Enrico > > Il giorno mer 15 dic 2021 alle ore 10:56 Haiting Jiang < > jianghait...@apache.org> ha scritto: > > > +1 (non-binding) > > > > Out of the scope of this PIP, a follow up data migration tool could be > > added to help migration from old settings without data loss. > > > > --- > > Haiting > > > > On 2021/12/15 01:04:21 Hang Chen wrote: > > > +1 > > > > > > Thanks, > > > Hang > > > > > > PengHui Li 于2021年12月15日周三 07:52写道： > > > > > > > > +1 > > > > > > > > Penghui > > > > > > > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli > > wrote: > > > > > > > > > https://github.com/apache/pulsar/issues/13302 > > > > > > > > > > Copying here for quoting convenience > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar > > cluster, > > > > > where > > > > > all the components are started within the context of a single JVM > > process. > > > > > > > > > > Users are using the standalone as a way to get quickly started with > > Pulsar > > > > > or > > > > > in all the cases where it makes sense to have a single node > > deployment. > > > > > > > > > > Right now, the standalone is starting by default with many > > components, > > > > > several of > > > > > which are quite complex, since they are designed to be deployed in a > > > > > distributed > > > > > fashion. > > > > > > > > > > ## Goal > > > > > > > > > > Simplify the components of Pulsar standalone to achieve: > > > > > > > > > > 1. Reduce complexity > > > > > 2. Reduce startup time > > > > > 3. Reduce memory and CPU footprint of running standalone > > > > > > > > > > ## Proposed changes > > > > > > > > > > The proposal here is to change some of the default implementations > > that are > > > > > used for the Pulsar standalone. > > > > > > > > > > 1. **Metadata Store implementation** --> > > > > > Change from ZooKeeper to RocksDB > > > > > > > > > > 2. **Pulsar functions package backend** --> > > > > > Change from using DistributedLog to using local filesystem, > > storing > > > > > the > > > > > jars directly in the data folder instead of uploading them > > into BK. > > > > > > > > > > 3. **Pulsar functions state store implementation** --> > > > > > Change the state store to be backed by a MetadataStore based > > backed, > > > > > with the RocksDB implementation. > > > > > > > > > > 4. **Table Service** --> > > > > > Do not start BK table service by default > > > > > > > > > > ## Compatibility considerations > > > > > > > > > > In order to avoid compatibility issues where users have existing > > Pulsar > > > > > standalone services that they want to upgrade without conflicts, we > > will > > > > > follow the principle of keeping the old defaults where there is > > existing > > > > > data on the disk. > > > > > > > > > > We will add a file, serving the purpose as a flag, in the > > `data/standalone` > > > > > directory, for example `new-2.10-defaults`. > > > > > > > > > > If the file is present, or if the data directory is completely > > missing, we > > > > > will adopt the new set of default configuration settings. > > > > > > > > > > If the file is not there, we will continue to use existing defaults > > and we > > > > > will > > > > > not break the upgrade operation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Matteo Merli > > > > > > > > > > > > > > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 @Haiting Pulsar Standalone is usually used for local development, I am not sure you need a migration tool. You can still keep the configuration to use previously (standalone.conf) Enrico Il giorno mer 15 dic 2021 alle ore 10:56 Haiting Jiang < jianghait...@apache.org> ha scritto: > +1 (non-binding) > > Out of the scope of this PIP, a follow up data migration tool could be > added to help migration from old settings without data loss. > > --- > Haiting > > On 2021/12/15 01:04:21 Hang Chen wrote: > > +1 > > > > Thanks, > > Hang > > > > PengHui Li 于2021年12月15日周三 07:52写道： > > > > > > +1 > > > > > > Penghui > > > > > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli > wrote: > > > > > > > https://github.com/apache/pulsar/issues/13302 > > > > > > > > Copying here for quoting convenience > > > > > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar > cluster, > > > > where > > > > all the components are started within the context of a single JVM > process. > > > > > > > > Users are using the standalone as a way to get quickly started with > Pulsar > > > > or > > > > in all the cases where it makes sense to have a single node > deployment. > > > > > > > > Right now, the standalone is starting by default with many > components, > > > > several of > > > > which are quite complex, since they are designed to be deployed in a > > > > distributed > > > > fashion. > > > > > > > > ## Goal > > > > > > > > Simplify the components of Pulsar standalone to achieve: > > > > > > > > 1. Reduce complexity > > > > 2. Reduce startup time > > > > 3. Reduce memory and CPU footprint of running standalone > > > > > > > > ## Proposed changes > > > > > > > > The proposal here is to change some of the default implementations > that are > > > > used for the Pulsar standalone. > > > > > > > > 1. **Metadata Store implementation** --> > > > > Change from ZooKeeper to RocksDB > > > > > > > > 2. **Pulsar functions package backend** --> > > > > Change from using DistributedLog to using local filesystem, > storing > > > > the > > > > jars directly in the data folder instead of uploading them > into BK. > > > > > > > > 3. **Pulsar functions state store implementation** --> > > > > Change the state store to be backed by a MetadataStore based > backed, > > > > with the RocksDB implementation. > > > > > > > > 4. **Table Service** --> > > > > Do not start BK table service by default > > > > > > > > ## Compatibility considerations > > > > > > > > In order to avoid compatibility issues where users have existing > Pulsar > > > > standalone services that they want to upgrade without conflicts, we > will > > > > follow the principle of keeping the old defaults where there is > existing > > > > data on the disk. > > > > > > > > We will add a file, serving the purpose as a flag, in the > `data/standalone` > > > > directory, for example `new-2.10-defaults`. > > > > > > > > If the file is present, or if the data directory is completely > missing, we > > > > will adopt the new set of default configuration settings. > > > > > > > > If the file is not there, we will continue to use existing defaults > and we > > > > will > > > > not break the upgrade operation. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Matteo Merli > > > > > > > > > > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 (non-binding) Out of the scope of this PIP, a follow up data migration tool could be added to help migration from old settings without data loss. --- Haiting On 2021/12/15 01:04:21 Hang Chen wrote: > +1 > > Thanks, > Hang > > PengHui Li 于2021年12月15日周三 07:52写道： > > > > +1 > > > > Penghui > > > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli wrote: > > > > > https://github.com/apache/pulsar/issues/13302 > > > > > > Copying here for quoting convenience > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, > > > where > > > all the components are started within the context of a single JVM process. > > > > > > Users are using the standalone as a way to get quickly started with Pulsar > > > or > > > in all the cases where it makes sense to have a single node deployment. > > > > > > Right now, the standalone is starting by default with many components, > > > several of > > > which are quite complex, since they are designed to be deployed in a > > > distributed > > > fashion. > > > > > > ## Goal > > > > > > Simplify the components of Pulsar standalone to achieve: > > > > > > 1. Reduce complexity > > > 2. Reduce startup time > > > 3. Reduce memory and CPU footprint of running standalone > > > > > > ## Proposed changes > > > > > > The proposal here is to change some of the default implementations that > > > are > > > used for the Pulsar standalone. > > > > > > 1. **Metadata Store implementation** --> > > > Change from ZooKeeper to RocksDB > > > > > > 2. **Pulsar functions package backend** --> > > > Change from using DistributedLog to using local filesystem, storing > > > the > > > jars directly in the data folder instead of uploading them into BK. > > > > > > 3. **Pulsar functions state store implementation** --> > > > Change the state store to be backed by a MetadataStore based backed, > > > with the RocksDB implementation. > > > > > > 4. **Table Service** --> > > > Do not start BK table service by default > > > > > > ## Compatibility considerations > > > > > > In order to avoid compatibility issues where users have existing Pulsar > > > standalone services that they want to upgrade without conflicts, we will > > > follow the principle of keeping the old defaults where there is existing > > > data on the disk. > > > > > > We will add a file, serving the purpose as a flag, in the > > > `data/standalone` > > > directory, for example `new-2.10-defaults`. > > > > > > If the file is present, or if the data directory is completely missing, we > > > will adopt the new set of default configuration settings. > > > > > > If the file is not there, we will continue to use existing defaults and we > > > will > > > not break the upgrade operation. > > > > > > > > > > > > > > > > > > -- > > > Matteo Merli > > > > > > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 (non-binding) On 2021/12/14 17:18:03 Matteo Merli wrote: > https://github.com/apache/pulsar/issues/13302 > > Copying here for quoting convenience > > > > > > ## Motivation > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, where > all the components are started within the context of a single JVM process. > > Users are using the standalone as a way to get quickly started with Pulsar or > in all the cases where it makes sense to have a single node deployment. > > Right now, the standalone is starting by default with many components, > several of > which are quite complex, since they are designed to be deployed in a > distributed > fashion. > > ## Goal > > Simplify the components of Pulsar standalone to achieve: > > 1. Reduce complexity > 2. Reduce startup time > 3. Reduce memory and CPU footprint of running standalone > > ## Proposed changes > > The proposal here is to change some of the default implementations that are > used for the Pulsar standalone. > > 1. **Metadata Store implementation** --> > Change from ZooKeeper to RocksDB > > 2. **Pulsar functions package backend** --> > Change from using DistributedLog to using local filesystem, storing the > jars directly in the data folder instead of uploading them into BK. > > 3. **Pulsar functions state store implementation** --> > Change the state store to be backed by a MetadataStore based backed, > with the RocksDB implementation. > > 4. **Table Service** --> > Do not start BK table service by default > > ## Compatibility considerations > > In order to avoid compatibility issues where users have existing Pulsar > standalone services that they want to upgrade without conflicts, we will > follow the principle of keeping the old defaults where there is existing > data on the disk. > > We will add a file, serving the purpose as a flag, in the `data/standalone` > directory, for example `new-2.10-defaults`. > > If the file is present, or if the data directory is completely missing, we > will adopt the new set of default configuration settings. > > If the file is not there, we will continue to use existing defaults and we > will > not break the upgrade operation. > > > > > > -- > Matteo Merli > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 Thanks, Hang PengHui Li 于2021年12月15日周三 07:52写道： > > +1 > > Penghui > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli wrote: > > > https://github.com/apache/pulsar/issues/13302 > > > > Copying here for quoting convenience > > > > > > > > > > > > ## Motivation > > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, > > where > > all the components are started within the context of a single JVM process. > > > > Users are using the standalone as a way to get quickly started with Pulsar > > or > > in all the cases where it makes sense to have a single node deployment. > > > > Right now, the standalone is starting by default with many components, > > several of > > which are quite complex, since they are designed to be deployed in a > > distributed > > fashion. > > > > ## Goal > > > > Simplify the components of Pulsar standalone to achieve: > > > > 1. Reduce complexity > > 2. Reduce startup time > > 3. Reduce memory and CPU footprint of running standalone > > > > ## Proposed changes > > > > The proposal here is to change some of the default implementations that are > > used for the Pulsar standalone. > > > > 1. **Metadata Store implementation** --> > > Change from ZooKeeper to RocksDB > > > > 2. **Pulsar functions package backend** --> > > Change from using DistributedLog to using local filesystem, storing > > the > > jars directly in the data folder instead of uploading them into BK. > > > > 3. **Pulsar functions state store implementation** --> > > Change the state store to be backed by a MetadataStore based backed, > > with the RocksDB implementation. > > > > 4. **Table Service** --> > > Do not start BK table service by default > > > > ## Compatibility considerations > > > > In order to avoid compatibility issues where users have existing Pulsar > > standalone services that they want to upgrade without conflicts, we will > > follow the principle of keeping the old defaults where there is existing > > data on the disk. > > > > We will add a file, serving the purpose as a flag, in the `data/standalone` > > directory, for example `new-2.10-defaults`. > > > > If the file is present, or if the data directory is completely missing, we > > will adopt the new set of default configuration settings. > > > > If the file is not there, we will continue to use existing defaults and we > > will > > not break the upgrade operation. > > > > > > > > > > > > -- > > Matteo Merli > > > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 Penghui On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli wrote: > https://github.com/apache/pulsar/issues/13302 > > Copying here for quoting convenience > > > > > > ## Motivation > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, > where > all the components are started within the context of a single JVM process. > > Users are using the standalone as a way to get quickly started with Pulsar > or > in all the cases where it makes sense to have a single node deployment. > > Right now, the standalone is starting by default with many components, > several of > which are quite complex, since they are designed to be deployed in a > distributed > fashion. > > ## Goal > > Simplify the components of Pulsar standalone to achieve: > > 1. Reduce complexity > 2. Reduce startup time > 3. Reduce memory and CPU footprint of running standalone > > ## Proposed changes > > The proposal here is to change some of the default implementations that are > used for the Pulsar standalone. > > 1. **Metadata Store implementation** --> > Change from ZooKeeper to RocksDB > > 2. **Pulsar functions package backend** --> > Change from using DistributedLog to using local filesystem, storing > the > jars directly in the data folder instead of uploading them into BK. > > 3. **Pulsar functions state store implementation** --> > Change the state store to be backed by a MetadataStore based backed, > with the RocksDB implementation. > > 4. **Table Service** --> > Do not start BK table service by default > > ## Compatibility considerations > > In order to avoid compatibility issues where users have existing Pulsar > standalone services that they want to upgrade without conflicts, we will > follow the principle of keeping the old defaults where there is existing > data on the disk. > > We will add a file, serving the purpose as a flag, in the `data/standalone` > directory, for example `new-2.10-defaults`. > > If the file is present, or if the data directory is completely missing, we > will adopt the new set of default configuration settings. > > If the file is not there, we will continue to use existing defaults and we > will > not break the upgrade operation. > > > > > > -- > Matteo Merli > >
[DISCUSSION] PIP-117: Change Pulsar standalone defaults
https://github.com/apache/pulsar/issues/13302 Copying here for quoting convenience ## Motivation Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, where all the components are started within the context of a single JVM process. Users are using the standalone as a way to get quickly started with Pulsar or in all the cases where it makes sense to have a single node deployment. Right now, the standalone is starting by default with many components, several of which are quite complex, since they are designed to be deployed in a distributed fashion. ## Goal Simplify the components of Pulsar standalone to achieve: 1. Reduce complexity 2. Reduce startup time 3. Reduce memory and CPU footprint of running standalone ## Proposed changes The proposal here is to change some of the default implementations that are used for the Pulsar standalone. 1. **Metadata Store implementation** --> Change from ZooKeeper to RocksDB 2. **Pulsar functions package backend** --> Change from using DistributedLog to using local filesystem, storing the jars directly in the data folder instead of uploading them into BK. 3. **Pulsar functions state store implementation** --> Change the state store to be backed by a MetadataStore based backed, with the RocksDB implementation. 4. **Table Service** --> Do not start BK table service by default ## Compatibility considerations In order to avoid compatibility issues where users have existing Pulsar standalone services that they want to upgrade without conflicts, we will follow the principle of keeping the old defaults where there is existing data on the disk. We will add a file, serving the purpose as a flag, in the `data/standalone` directory, for example `new-2.10-defaults`. If the file is present, or if the data directory is completely missing, we will adopt the new set of default configuration settings. If the file is not there, we will continue to use existing defaults and we will not break the upgrade operation. -- Matteo Merli
