Re: Moving binary metadata to PDS storage folder
Hello everyone, can we move forward with the merge of IGNITE-12994? Semyon 18.05.2020, 19:08, "Denis Mekhanikov" : > Users shouldn't care about binary_meta and marshaller directories. Those > are internal details. Having to go through the documentation on those > things to figure out whether you need to run each of the provided scripts > or not will lead to terrible user experience. > If we add the binary_marshaller_move.sh script for this specific case, it > will stay with us forever just because migration from Ignite 2.8 to 2.9 > required running this thing. > > Denis
Re: Moving binary metadata to PDS storage folder
Users shouldn't care about binary_meta and marshaller directories. Those are internal details. Having to go through the documentation on those things to figure out whether you need to run each of the provided scripts or not will lead to terrible user experience. If we add the binary_marshaller_move.sh script for this specific case, it will stay with us forever just because migration from Ignite 2.8 to 2.9 required running this thing. Denis
Re: Moving binary metadata to PDS storage folder
Folks, I don't agree that it should be a single script for the whole migration process. I don't think we should have some kind of automated `decision-maker` processes for the migration procedure. We should have well-defined documentation pages + the `migration` directory with a single .sh per single operation. For example: 1. binary_marshaller_move.sh -- move to binary, marshaller directories. 2. nodeId_to_consistentId.sh -- change a node directory to the new format. The migration for each new version will require different sets of these scripts (supposedly less than 10 in total). I think users will be able to run each script manually to achieve PDS compatibility according to their version. On Fri, 15 May 2020 at 17:19, Anton Kalashnikov wrote: > > Hello, > > I agree with the described disadvantages(which Maxim mentioned) and I think > it is a good idea to discuss a new way of pds migration in an extra ticket. > But also, I agree with Denis that the script will add ambiguity(it's not > clear which script should be used and when) if we add it. In my opinion, > maybe the better way is to store some migration metainformation in ignite > metastorage, and in case, when this information not found, execute some > migration code - but of course, there is a lot to discuss. > > According to this task, I don't see any problem if we left it as is because > the overhead is minimum right now(only on start and only if the old folder > exists). Also, we can easily drop(or move) this piece of code if we implement > a new migration way in some time. > > -- > Best regards, > Anton Kalashnikov > > > 13.05.2020, 21:59, "Denis Mekhanikov" : > > Maxim, > > > > This way we'll introduce a migration procedure between versions, which > > we currently don't have. Different Ignite versions are compatible by > > persistence storage. If we add a migration script, we need to decide, > > whether we need to run it every time when the version is upgraded, or > > only some specific versions are affected. > > > > I suggest having a procedure that will look for metadata in the work > > directory, and if it finds it there, then the node will use it. > > Otherwise the persistence directory is used. > > > > Denis > > > > On 13.05.2020 21:40, Maxim Muzafarov wrote: > >> Folks, > >> > >> I think it's important to discuss the following question regarding this > >> thread: > >> Should we consider moving the migration procedure from the java > >> production code to migration scripts? > >> > >> From my understanding, keeping all such things in java production > >> source code has some disadvantages: > >> 1. It executes only once at the migration stage. > >> 2. It affects the complexity of the source code and code maintenance. > >> 3. Node crash cases must be covered during the migration procedure. > >> 4. It affects the production usage e.g. the process doesn't have the > >> right access to the old directory (migration already completed) and > >> will fail the node start. > >> > >> The right behavior from my point should be: > >> 1. Change the default path of binary/marshaller directory to the new one. > >> 2. Provide migration scripts for users. > >> > >> WDYT? > >> > >> On Wed, 13 May 2020 at 21:10, Denis Mekhanikov > >> wrote: > >>> Sounds great! > >>> > >>> It happens pretty frequently that users migrate to a new version of > >>> Ignite and preserve persistence files only without caring too much about > >>> the work folder. But it turns out, that the work folder actually has > >>> some important stuff. > >>> This improvement should help with this issue. > >>> > >>> What's about in-memory mode? As far as I know, we write binary metadata > >>> to disk even when no persistence is configured. Do you plan to address > >>> it in any way? > >>> > >>> Denis > >>> > >>> On 12.05.2020 15:56, Sergey Antonov wrote: > Hello Semyon, > > This is a good idea! > > вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин > : > > > Hello Semyon, > > > > This is a good and long-awaited improvement! Thank you for your > > efforts! > > > > Thanks, > > S. > > > > вт, 12 мая 2020 г. в 15:11, Данилов Семён : > > > >> Hello! > >> > >> I would like to propose moving /binary_meta and /marshaller folders to > > the > >> PDS folder. > >> > >> Motivation: data, directly related to the persistence, is stored > >> outside > >> the persistence dir, which can lead to various issues and also is not > > very > >> convenient to use. In particular, with k8s, deployment disk that is > >> attached to a container can not be accessed from other containers or > >> outside of k8s. In case if support will need to drop persistence > >> except > >> data, there will be no way to recover due to binary metadata is > >> required > > to > >> process PDS files. > >> > >> I created an
Re: Moving binary metadata to PDS storage folder
Hello, I agree with the described disadvantages(which Maxim mentioned) and I think it is a good idea to discuss a new way of pds migration in an extra ticket. But also, I agree with Denis that the script will add ambiguity(it's not clear which script should be used and when) if we add it. In my opinion, maybe the better way is to store some migration metainformation in ignite metastorage, and in case, when this information not found, execute some migration code - but of course, there is a lot to discuss. According to this task, I don't see any problem if we left it as is because the overhead is minimum right now(only on start and only if the old folder exists). Also, we can easily drop(or move) this piece of code if we implement a new migration way in some time. -- Best regards, Anton Kalashnikov 13.05.2020, 21:59, "Denis Mekhanikov" : > Maxim, > > This way we'll introduce a migration procedure between versions, which > we currently don't have. Different Ignite versions are compatible by > persistence storage. If we add a migration script, we need to decide, > whether we need to run it every time when the version is upgraded, or > only some specific versions are affected. > > I suggest having a procedure that will look for metadata in the work > directory, and if it finds it there, then the node will use it. > Otherwise the persistence directory is used. > > Denis > > On 13.05.2020 21:40, Maxim Muzafarov wrote: >> Folks, >> >> I think it's important to discuss the following question regarding this >> thread: >> Should we consider moving the migration procedure from the java >> production code to migration scripts? >> >> From my understanding, keeping all such things in java production >> source code has some disadvantages: >> 1. It executes only once at the migration stage. >> 2. It affects the complexity of the source code and code maintenance. >> 3. Node crash cases must be covered during the migration procedure. >> 4. It affects the production usage e.g. the process doesn't have the >> right access to the old directory (migration already completed) and >> will fail the node start. >> >> The right behavior from my point should be: >> 1. Change the default path of binary/marshaller directory to the new one. >> 2. Provide migration scripts for users. >> >> WDYT? >> >> On Wed, 13 May 2020 at 21:10, Denis Mekhanikov >> wrote: >>> Sounds great! >>> >>> It happens pretty frequently that users migrate to a new version of >>> Ignite and preserve persistence files only without caring too much about >>> the work folder. But it turns out, that the work folder actually has >>> some important stuff. >>> This improvement should help with this issue. >>> >>> What's about in-memory mode? As far as I know, we write binary metadata >>> to disk even when no persistence is configured. Do you plan to address >>> it in any way? >>> >>> Denis >>> >>> On 12.05.2020 15:56, Sergey Antonov wrote: Hello Semyon, This is a good idea! вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : > Hello Semyon, > > This is a good and long-awaited improvement! Thank you for your efforts! > > Thanks, > S. > > вт, 12 мая 2020 г. в 15:11, Данилов Семён : > >> Hello! >> >> I would like to propose moving /binary_meta and /marshaller folders to > the >> PDS folder. >> >> Motivation: data, directly related to the persistence, is stored outside >> the persistence dir, which can lead to various issues and also is not > very >> convenient to use. In particular, with k8s, deployment disk that is >> attached to a container can not be accessed from other containers or >> outside of k8s. In case if support will need to drop persistence except >> data, there will be no way to recover due to binary metadata is required > to >> process PDS files. >> >> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) > and a >> pull request(https://github.com/apache/ignite/pull/7792) that fixes the >> issue. >> >> In that PR I made the following: >> >> * store binary meta and marshaller data inside db/ folder >> * if binary meta of marshaller are found in "legacy" locations -- >> safely move them to new locations during the node startup >> >> Kind regards, >> >> Semyon Danilov.
Re: Moving binary metadata to PDS storage folder
Hello Denis, I found out that without persistence being enabled, no metadata is written on disk (where is an if clause at the start of every method of BinaryMetadataFileStore) and everything is stored in metadataLocalCache. Semyon. 13.05.2020, 21:10, "Denis Mekhanikov" : > Sounds great! > > It happens pretty frequently that users migrate to a new version of > Ignite and preserve persistence files only without caring too much about > the work folder. But it turns out, that the work folder actually has > some important stuff. > This improvement should help with this issue. > > What's about in-memory mode? As far as I know, we write binary metadata > to disk even when no persistence is configured. Do you plan to address > it in any way? > > Denis > > On 12.05.2020 15:56, Sergey Antonov wrote: >> Hello Semyon, >> >> This is a good idea! >> >> вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : >> >>> Hello Semyon, >>> >>> This is a good and long-awaited improvement! Thank you for your efforts! >>> >>> Thanks, >>> S. >>> >>> вт, 12 мая 2020 г. в 15:11, Данилов Семён : >>> Hello! I would like to propose moving /binary_meta and /marshaller folders to >>> the PDS folder. Motivation: data, directly related to the persistence, is stored outside the persistence dir, which can lead to various issues and also is not >>> very convenient to use. In particular, with k8s, deployment disk that is attached to a container can not be accessed from other containers or outside of k8s. In case if support will need to drop persistence except data, there will be no way to recover due to binary metadata is required >>> to process PDS files. I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) >>> and a pull request(https://github.com/apache/ignite/pull/7792) that fixes the issue. In that PR I made the following: * store binary meta and marshaller data inside db/ folder * if binary meta of marshaller are found in "legacy" locations -- safely move them to new locations during the node startup Kind regards, Semyon Danilov.
Re: Moving binary metadata to PDS storage folder
Maxim, This way we'll introduce a migration procedure between versions, which we currently don't have. Different Ignite versions are compatible by persistence storage. If we add a migration script, we need to decide, whether we need to run it every time when the version is upgraded, or only some specific versions are affected. I suggest having a procedure that will look for metadata in the work directory, and if it finds it there, then the node will use it. Otherwise the persistence directory is used. Denis On 13.05.2020 21:40, Maxim Muzafarov wrote: Folks, I think it's important to discuss the following question regarding this thread: Should we consider moving the migration procedure from the java production code to migration scripts? From my understanding, keeping all such things in java production source code has some disadvantages: 1. It executes only once at the migration stage. 2. It affects the complexity of the source code and code maintenance. 3. Node crash cases must be covered during the migration procedure. 4. It affects the production usage e.g. the process doesn't have the right access to the old directory (migration already completed) and will fail the node start. The right behavior from my point should be: 1. Change the default path of binary/marshaller directory to the new one. 2. Provide migration scripts for users. WDYT? On Wed, 13 May 2020 at 21:10, Denis Mekhanikov wrote: Sounds great! It happens pretty frequently that users migrate to a new version of Ignite and preserve persistence files only without caring too much about the work folder. But it turns out, that the work folder actually has some important stuff. This improvement should help with this issue. What's about in-memory mode? As far as I know, we write binary metadata to disk even when no persistence is configured. Do you plan to address it in any way? Denis On 12.05.2020 15:56, Sergey Antonov wrote: Hello Semyon, This is a good idea! вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : Hello Semyon, This is a good and long-awaited improvement! Thank you for your efforts! Thanks, S. вт, 12 мая 2020 г. в 15:11, Данилов Семён : Hello! I would like to propose moving /binary_meta and /marshaller folders to the PDS folder. Motivation: data, directly related to the persistence, is stored outside the persistence dir, which can lead to various issues and also is not very convenient to use. In particular, with k8s, deployment disk that is attached to a container can not be accessed from other containers or outside of k8s. In case if support will need to drop persistence except data, there will be no way to recover due to binary metadata is required to process PDS files. I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a pull request(https://github.com/apache/ignite/pull/7792) that fixes the issue. In that PR I made the following: * store binary meta and marshaller data inside db/ folder * if binary meta of marshaller are found in "legacy" locations -- safely move them to new locations during the node startup Kind regards, Semyon Danilov.
Re: Moving binary metadata to PDS storage folder
Folks, I think it's important to discuss the following question regarding this thread: Should we consider moving the migration procedure from the java production code to migration scripts? >From my understanding, keeping all such things in java production source code has some disadvantages: 1. It executes only once at the migration stage. 2. It affects the complexity of the source code and code maintenance. 3. Node crash cases must be covered during the migration procedure. 4. It affects the production usage e.g. the process doesn't have the right access to the old directory (migration already completed) and will fail the node start. The right behavior from my point should be: 1. Change the default path of binary/marshaller directory to the new one. 2. Provide migration scripts for users. WDYT? On Wed, 13 May 2020 at 21:10, Denis Mekhanikov wrote: > > Sounds great! > > It happens pretty frequently that users migrate to a new version of > Ignite and preserve persistence files only without caring too much about > the work folder. But it turns out, that the work folder actually has > some important stuff. > This improvement should help with this issue. > > What's about in-memory mode? As far as I know, we write binary metadata > to disk even when no persistence is configured. Do you plan to address > it in any way? > > Denis > > On 12.05.2020 15:56, Sergey Antonov wrote: > > Hello Semyon, > > > > This is a good idea! > > > > вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : > > > >> Hello Semyon, > >> > >> This is a good and long-awaited improvement! Thank you for your efforts! > >> > >> Thanks, > >> S. > >> > >> вт, 12 мая 2020 г. в 15:11, Данилов Семён : > >> > >>> Hello! > >>> > >>> I would like to propose moving /binary_meta and /marshaller folders to > >> the > >>> PDS folder. > >>> > >>> Motivation: data, directly related to the persistence, is stored outside > >>> the persistence dir, which can lead to various issues and also is not > >> very > >>> convenient to use. In particular, with k8s, deployment disk that is > >>> attached to a container can not be accessed from other containers or > >>> outside of k8s. In case if support will need to drop persistence except > >>> data, there will be no way to recover due to binary metadata is required > >> to > >>> process PDS files. > >>> > >>> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) > >> and a > >>> pull request(https://github.com/apache/ignite/pull/7792) that fixes the > >>> issue. > >>> > >>> In that PR I made the following: > >>> > >>> > >>> * store binary meta and marshaller data inside db/ folder > >>> * if binary meta of marshaller are found in "legacy" locations -- > >>> safely move them to new locations during the node startup > >>> > >>> > >>> Kind regards, > >>> > >>> Semyon Danilov. > >>> > >
Re: Moving binary metadata to PDS storage folder
Sounds great! It happens pretty frequently that users migrate to a new version of Ignite and preserve persistence files only without caring too much about the work folder. But it turns out, that the work folder actually has some important stuff. This improvement should help with this issue. What's about in-memory mode? As far as I know, we write binary metadata to disk even when no persistence is configured. Do you plan to address it in any way? Denis On 12.05.2020 15:56, Sergey Antonov wrote: Hello Semyon, This is a good idea! вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : Hello Semyon, This is a good and long-awaited improvement! Thank you for your efforts! Thanks, S. вт, 12 мая 2020 г. в 15:11, Данилов Семён : Hello! I would like to propose moving /binary_meta and /marshaller folders to the PDS folder. Motivation: data, directly related to the persistence, is stored outside the persistence dir, which can lead to various issues and also is not very convenient to use. In particular, with k8s, deployment disk that is attached to a container can not be accessed from other containers or outside of k8s. In case if support will need to drop persistence except data, there will be no way to recover due to binary metadata is required to process PDS files. I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a pull request(https://github.com/apache/ignite/pull/7792) that fixes the issue. In that PR I made the following: * store binary meta and marshaller data inside db/ folder * if binary meta of marshaller are found in "legacy" locations -- safely move them to new locations during the node startup Kind regards, Semyon Danilov.
Re: Moving binary metadata to PDS storage folder
Hello Semyon, This is a good idea! вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : > Hello Semyon, > > This is a good and long-awaited improvement! Thank you for your efforts! > > Thanks, > S. > > вт, 12 мая 2020 г. в 15:11, Данилов Семён : > > > Hello! > > > > I would like to propose moving /binary_meta and /marshaller folders to > the > > PDS folder. > > > > Motivation: data, directly related to the persistence, is stored outside > > the persistence dir, which can lead to various issues and also is not > very > > convenient to use. In particular, with k8s, deployment disk that is > > attached to a container can not be accessed from other containers or > > outside of k8s. In case if support will need to drop persistence except > > data, there will be no way to recover due to binary metadata is required > to > > process PDS files. > > > > I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) > and a > > pull request(https://github.com/apache/ignite/pull/7792) that fixes the > > issue. > > > > In that PR I made the following: > > > > > > * store binary meta and marshaller data inside db/ folder > > * if binary meta of marshaller are found in "legacy" locations -- > > safely move them to new locations during the node startup > > > > > > Kind regards, > > > > Semyon Danilov. > > > -- BR, Sergey Antonov
Re: Moving binary metadata to PDS storage folder
Hello Semyon, This is a good and long-awaited improvement! Thank you for your efforts! Thanks, S. вт, 12 мая 2020 г. в 15:11, Данилов Семён : > Hello! > > I would like to propose moving /binary_meta and /marshaller folders to the > PDS folder. > > Motivation: data, directly related to the persistence, is stored outside > the persistence dir, which can lead to various issues and also is not very > convenient to use. In particular, with k8s, deployment disk that is > attached to a container can not be accessed from other containers or > outside of k8s. In case if support will need to drop persistence except > data, there will be no way to recover due to binary metadata is required to > process PDS files. > > I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a > pull request(https://github.com/apache/ignite/pull/7792) that fixes the > issue. > > In that PR I made the following: > > > * store binary meta and marshaller data inside db/ folder > * if binary meta of marshaller are found in "legacy" locations -- > safely move them to new locations during the node startup > > > Kind regards, > > Semyon Danilov. >
Moving binary metadata to PDS storage folder
Hello! I would like to propose moving /binary_meta and /marshaller folders to the PDS folder. Motivation: data, directly related to the persistence, is stored outside the persistence dir, which can lead to various issues and also is not very convenient to use. In particular, with k8s, deployment disk that is attached to a container can not be accessed from other containers or outside of k8s. In case if support will need to drop persistence except data, there will be no way to recover due to binary metadata is required to process PDS files. I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a pull request(https://github.com/apache/ignite/pull/7792) that fixes the issue. In that PR I made the following: * store binary meta and marshaller data inside db/ folder * if binary meta of marshaller are found in "legacy" locations -- safely move them to new locations during the node startup Kind regards, Semyon Danilov.
