Hello, I agree with the described disadvantages(which Maxim mentioned) and I think it is a good idea to discuss a new way of pds migration in an extra ticket. But also, I agree with Denis that the script will add ambiguity(it's not clear which script should be used and when) if we add it. In my opinion, maybe the better way is to store some migration metainformation in ignite metastorage, and in case, when this information not found, execute some migration code - but of course, there is a lot to discuss.
According to this task, I don't see any problem if we left it as is because the overhead is minimum right now(only on start and only if the old folder exists). Also, we can easily drop(or move) this piece of code if we implement a new migration way in some time. -- Best regards, Anton Kalashnikov 13.05.2020, 21:59, "Denis Mekhanikov" <dmekhani...@gmail.com>: > Maxim, > > This way we'll introduce a migration procedure between versions, which > we currently don't have. Different Ignite versions are compatible by > persistence storage. If we add a migration script, we need to decide, > whether we need to run it every time when the version is upgraded, or > only some specific versions are affected. > > I suggest having a procedure that will look for metadata in the work > directory, and if it finds it there, then the node will use it. > Otherwise the persistence directory is used. > > Denis > > On 13.05.2020 21:40, Maxim Muzafarov wrote: >> Folks, >> >> I think it's important to discuss the following question regarding this >> thread: >> Should we consider moving the migration procedure from the java >> production code to migration scripts? >> >> From my understanding, keeping all such things in java production >> source code has some disadvantages: >> 1. It executes only once at the migration stage. >> 2. It affects the complexity of the source code and code maintenance. >> 3. Node crash cases must be covered during the migration procedure. >> 4. It affects the production usage e.g. the process doesn't have the >> right access to the old directory (migration already completed) and >> will fail the node start. >> >> The right behavior from my point should be: >> 1. Change the default path of binary/marshaller directory to the new one. >> 2. Provide migration scripts for users. >> >> WDYT? >> >> On Wed, 13 May 2020 at 21:10, Denis Mekhanikov <dmekhani...@gmail.com> >> wrote: >>> Sounds great! >>> >>> It happens pretty frequently that users migrate to a new version of >>> Ignite and preserve persistence files only without caring too much about >>> the work folder. But it turns out, that the work folder actually has >>> some important stuff. >>> This improvement should help with this issue. >>> >>> What's about in-memory mode? As far as I know, we write binary metadata >>> to disk even when no persistence is configured. Do you plan to address >>> it in any way? >>> >>> Denis >>> >>> On 12.05.2020 15:56, Sergey Antonov wrote: >>>> Hello Semyon, >>>> >>>> This is a good idea! >>>> >>>> вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин <slava.kopti...@gmail.com>: >>>> >>>>> Hello Semyon, >>>>> >>>>> This is a good and long-awaited improvement! Thank you for your efforts! >>>>> >>>>> Thanks, >>>>> S. >>>>> >>>>> вт, 12 мая 2020 г. в 15:11, Данилов Семён <samvi...@yandex.ru>: >>>>> >>>>>> Hello! >>>>>> >>>>>> I would like to propose moving /binary_meta and /marshaller folders to >>>>> the >>>>>> PDS folder. >>>>>> >>>>>> Motivation: data, directly related to the persistence, is stored outside >>>>>> the persistence dir, which can lead to various issues and also is not >>>>> very >>>>>> convenient to use. In particular, with k8s, deployment disk that is >>>>>> attached to a container can not be accessed from other containers or >>>>>> outside of k8s. In case if support will need to drop persistence except >>>>>> data, there will be no way to recover due to binary metadata is required >>>>> to >>>>>> process PDS files. >>>>>> >>>>>> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) >>>>> and a >>>>>> pull request(https://github.com/apache/ignite/pull/7792) that fixes the >>>>>> issue. >>>>>> >>>>>> In that PR I made the following: >>>>>> >>>>>> * store binary meta and marshaller data inside db/ folder >>>>>> * if binary meta of marshaller are found in "legacy" locations -- >>>>>> safely move them to new locations during the node startup >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Semyon Danilov.
