Folks,
I don't agree that it should be a single script for the whole migration process. I don't think we should have some kind of automated `decision-maker` processes for the migration procedure. We should have well-defined documentation pages + the `migration` directory with a single .sh per single operation. For example: 1. binary_marshaller_move.sh -- move to binary, marshaller directories. 2. nodeId_to_consistentId.sh -- change a node directory to the new format. The migration for each new version will require different sets of these scripts (supposedly less than 10 in total). I think users will be able to run each script manually to achieve PDS compatibility according to their version. On Fri, 15 May 2020 at 17:19, Anton Kalashnikov <kaa....@yandex.ru> wrote: > > Hello, > > I agree with the described disadvantages(which Maxim mentioned) and I think > it is a good idea to discuss a new way of pds migration in an extra ticket. > But also, I agree with Denis that the script will add ambiguity(it's not > clear which script should be used and when) if we add it. In my opinion, > maybe the better way is to store some migration metainformation in ignite > metastorage, and in case, when this information not found, execute some > migration code - but of course, there is a lot to discuss. > > According to this task, I don't see any problem if we left it as is because > the overhead is minimum right now(only on start and only if the old folder > exists). Also, we can easily drop(or move) this piece of code if we implement > a new migration way in some time. > > -- > Best regards, > Anton Kalashnikov > > > 13.05.2020, 21:59, "Denis Mekhanikov" <dmekhani...@gmail.com>: > > Maxim, > > > > This way we'll introduce a migration procedure between versions, which > > we currently don't have. Different Ignite versions are compatible by > > persistence storage. If we add a migration script, we need to decide, > > whether we need to run it every time when the version is upgraded, or > > only some specific versions are affected. > > > > I suggest having a procedure that will look for metadata in the work > > directory, and if it finds it there, then the node will use it. > > Otherwise the persistence directory is used. > > > > Denis > > > > On 13.05.2020 21:40, Maxim Muzafarov wrote: > >> Folks, > >> > >> I think it's important to discuss the following question regarding this > >> thread: > >> Should we consider moving the migration procedure from the java > >> production code to migration scripts? > >> > >> From my understanding, keeping all such things in java production > >> source code has some disadvantages: > >> 1. It executes only once at the migration stage. > >> 2. It affects the complexity of the source code and code maintenance. > >> 3. Node crash cases must be covered during the migration procedure. > >> 4. It affects the production usage e.g. the process doesn't have the > >> right access to the old directory (migration already completed) and > >> will fail the node start. > >> > >> The right behavior from my point should be: > >> 1. Change the default path of binary/marshaller directory to the new one. > >> 2. Provide migration scripts for users. > >> > >> WDYT? > >> > >> On Wed, 13 May 2020 at 21:10, Denis Mekhanikov <dmekhani...@gmail.com> > >> wrote: > >>> Sounds great! > >>> > >>> It happens pretty frequently that users migrate to a new version of > >>> Ignite and preserve persistence files only without caring too much about > >>> the work folder. But it turns out, that the work folder actually has > >>> some important stuff. > >>> This improvement should help with this issue. > >>> > >>> What's about in-memory mode? As far as I know, we write binary metadata > >>> to disk even when no persistence is configured. Do you plan to address > >>> it in any way? > >>> > >>> Denis > >>> > >>> On 12.05.2020 15:56, Sergey Antonov wrote: > >>>> Hello Semyon, > >>>> > >>>> This is a good idea! > >>>> > >>>> вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин > >>>> <slava.kopti...@gmail.com>: > >>>> > >>>>> Hello Semyon, > >>>>> > >>>>> This is a good and long-awaited improvement! Thank you for your > >>>>> efforts! > >>>>> > >>>>> Thanks, > >>>>> S. > >>>>> > >>>>> вт, 12 мая 2020 г. в 15:11, Данилов Семён <samvi...@yandex.ru>: > >>>>> > >>>>>> Hello! > >>>>>> > >>>>>> I would like to propose moving /binary_meta and /marshaller folders to > >>>>> the > >>>>>> PDS folder. > >>>>>> > >>>>>> Motivation: data, directly related to the persistence, is stored > >>>>>> outside > >>>>>> the persistence dir, which can lead to various issues and also is not > >>>>> very > >>>>>> convenient to use. In particular, with k8s, deployment disk that is > >>>>>> attached to a container can not be accessed from other containers or > >>>>>> outside of k8s. In case if support will need to drop persistence > >>>>>> except > >>>>>> data, there will be no way to recover due to binary metadata is > >>>>>> required > >>>>> to > >>>>>> process PDS files. > >>>>>> > >>>>>> I created an issue > >>>>>> (https://issues.apache.org/jira/browse/IGNITE-12994) > >>>>> and a > >>>>>> pull request(https://github.com/apache/ignite/pull/7792) that fixes > >>>>>> the > >>>>>> issue. > >>>>>> > >>>>>> In that PR I made the following: > >>>>>> > >>>>>> * store binary meta and marshaller data inside db/ folder > >>>>>> * if binary meta of marshaller are found in "legacy" locations -- > >>>>>> safely move them to new locations during the node startup > >>>>>> > >>>>>> Kind regards, > >>>>>> > >>>>>> Semyon Danilov.
