Hi Enrico, This migration tool is a better-to-have and mostly about metadata store part.
Some of our user use Standalone in a long running test environment, with this migration tool they do not need to recreate all the topics and subscriptions manually and take these benefits with new default settings. And furthermore, since we have different kinds of metadata store implementations, metadata migration is a potential requirement. Just some brief thoughts yet. I think we can start a new DISCUSSION on this. Thanks, Haiting On 2021/12/15 11:09:22 Enrico Olivelli wrote: > +1 > > @Haiting > Pulsar Standalone is usually used for local development, I am not sure you > need a migration tool. > You can still keep the configuration to use previously (standalone.conf) > > Enrico > > Il giorno mer 15 dic 2021 alle ore 10:56 Haiting Jiang < > jianghait...@apache.org> ha scritto: > > > +1 (non-binding) > > > > Out of the scope of this PIP, a follow up data migration tool could be > > added to help migration from old settings without data loss. > > > > --- > > Haiting > > > > On 2021/12/15 01:04:21 Hang Chen wrote: > > > +1 > > > > > > Thanks, > > > Hang > > > > > > PengHui Li <peng...@apache.org> 于2021年12月15日周三 07:52写道： > > > > > > > > +1 > > > > > > > > Penghui > > > > > > > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli <mme...@apache.org> > > wrote: > > > > > > > > > https://github.com/apache/pulsar/issues/13302 > > > > > > > > > > Copying here for quoting convenience > > > > > ---- > > > > > > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar > > cluster, > > > > > where > > > > > all the components are started within the context of a single JVM > > process. > > > > > > > > > > Users are using the standalone as a way to get quickly started with > > Pulsar > > > > > or > > > > > in all the cases where it makes sense to have a single node > > deployment. > > > > > > > > > > Right now, the standalone is starting by default with many > > components, > > > > > several of > > > > > which are quite complex, since they are designed to be deployed in a > > > > > distributed > > > > > fashion. > > > > > > > > > > ## Goal > > > > > > > > > > Simplify the components of Pulsar standalone to achieve: > > > > > > > > > > 1. Reduce complexity > > > > > 2. Reduce startup time > > > > > 3. Reduce memory and CPU footprint of running standalone > > > > > > > > > > ## Proposed changes > > > > > > > > > > The proposal here is to change some of the default implementations > > that are > > > > > used for the Pulsar standalone. > > > > > > > > > > 1. **Metadata Store implementation** --> > > > > > Change from ZooKeeper to RocksDB > > > > > > > > > > 2. **Pulsar functions package backend** --> > > > > > Change from using DistributedLog to using local filesystem, > > storing > > > > > the > > > > > jars directly in the data folder instead of uploading them > > into BK. > > > > > > > > > > 3. **Pulsar functions state store implementation** --> > > > > > Change the state store to be backed by a MetadataStore based > > backed, > > > > > with the RocksDB implementation. > > > > > > > > > > 4. **Table Service** --> > > > > > Do not start BK table service by default > > > > > > > > > > ## Compatibility considerations > > > > > > > > > > In order to avoid compatibility issues where users have existing > > Pulsar > > > > > standalone services that they want to upgrade without conflicts, we > > will > > > > > follow the principle of keeping the old defaults where there is > > existing > > > > > data on the disk. > > > > > > > > > > We will add a file, serving the purpose as a flag, in the > > `data/standalone` > > > > > directory, for example `new-2.10-defaults`. > > > > > > > > > > If the file is present, or if the data directory is completely > > missing, we > > > > > will adopt the new set of default configuration settings. > > > > > > > > > > If the file is not there, we will continue to use existing defaults > > and we > > > > > will > > > > > not break the upgrade operation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Matteo Merli > > > > > <mme...@apache.org> > > > > > > > > > > >
