Hello. > If we want to make metastore more reliable we should provide some command to > remove keys from it.
Ticket [1] to provide this feature created. [1] https://issues.apache.org/jira/browse/IGNITE-13901 > 22 дек. 2020 г., в 17:57, Nikolay Izhikov <nizhikov....@gmail.com> написал(а): > > Hello, Anton. > > Thanks for the answer! > >> Only maybe, it makes sense to remove such records (with all history) instead >> of filtering them. > > I don’t think we should do this, by default. > > Imagine scenario #1: > > 1. Update the plugin to the next bugged version. Some class that is written > to the metastore is lost. > 2. Remove the key from the metastore to fix current deployment. > 3. Deploy updated plugin > > We will lose plugin information from metastore on step #3. > > If we want to make metastore more reliable we should provide some command to > remove keys from it. > This will fix scenario #2 when we migrate from one Ignite deployment to > another and some keys become unused. > >> 22 дек. 2020 г., в 17:39, Anton Kalashnikov <kaa....@yandex.ru> написал(а): >> >>> How it differs from the current implementation? >> There are no differences in implementation. But according to the original >> topic, plugins or other external things can write to metastore their own >> classes. I just said that according to current architecture it is not a good >> idea to do that because these POJO's are not from the core. >> >>> Why do you think that «good» solution should exist for this kind of issue? >> I don't think so. I just emphasize my concern about local filtering(the >> usage of system property) in this solution because it can lead to different >> behavior on different nodes. But perhaps you are right, and for such bug we >> can use a such solution. >> >>> Maybe we should make metastore fully lazy, so any stored key will not be >>> deserialized before it explicitly queried. >> I approximately meant the same. We should think about that. >> >> In conclusion(my opinions): >> >> In the current design, inside of plugins(etc.) it makes sense to use only >> primitives or POJO's from the core. (This is my answer to the original topic) >> It makes sense to think about serialization by demand inside of metastorage >> rather than in the discovery. >> I have no good solution for resolving the bug with removed classes. Perhaps, >> we can use Nikolay's fix. Only maybe, it makes sense to remove such records >> (with all history) instead of filtering them. >> >> -- >> Best regards, >> Anton Kalashnikov >> >> >> >> 22.12.2020, 14:44, "Nikolay Izhikov" <nizhi...@apache.org>: >>> Hello, Anton. >>> >>>> or use the POJO from the ignite core. >>> >>> How it differs from the current implementation? >>> >>>> As I can see you have tests only for one node but what happens if >>>> different nodes have different filters? >>> >>> The error will happen. >>> >>> Please, don’t forget that we are talking about two scenarios: >>> >>> 1. Blocker bug - we delete some class that was written to metastore from >>> the new version. >>> 2. Migration between custom Ignite distributions where one of them has a >>> custom class and the other doesn’t. >>> >>> It’s a very rare incident in my experience. >>> >>> Why do you think that «good» solution should exist for this kind of issue? >>> I don’t think we should limit internal architecture to cover these cases. >>> >>> Maybe we should make metastore fully lazy, so any stored key will not be >>> deserialized before it explicitly queried. >>> >>>> 22 дек. 2020 г., в 14:30, Anton Kalashnikov <kaa....@yandex.ru> написал(а): >>>> >>>> Hello everyone, >>>> >>>> In my opinion, it is indeed better to limit storing to the metastore by >>>> primitive type(map or list are also possible) or use the POJO from the >>>> ignite core. >>>> As Kirill correctly notice, right now, it is a problem not inside of the >>>> distributed metastore but inside of discovery. >>>> In fact, we can rewrite the sending metastore data in such a way that the >>>> discovery would think that there is just a simple byte array which >>>> shouldn't be deserialized. Right now, it understands that it is a >>>> serialized java object and it tries to deserialize it after receiving it. >>>> But this way requires more investigation about possible corner cases. >>>> >>>> Nikolay, I also don't sure that your fix handles metastorage history >>>> correctly. >>>> As I can see you have tests only for one node but what happens if >>>> different nodes have different filters? >>>> or if we need to send history to the joining node but some of the keys >>>> don't pass the filter? >>>> Maybe I wrong but in the first eye, it can lead to different >>>> results/histories on different nodes which is a problem. >>>> I just briefly looked at your PR(so maybe I didn't understand something), >>>> I will try to do it more carefully at the nearest time. >>>> >>>> -- >>>> Best regards, >>>> Anton Kalashnikov >>>> >>>> 18.12.2020, 15:33, "Mekhanikov Denis" <dmekhani...@gmail.com>: >>>>> Nikolay, >>>>> >>>>> Thanks for your reply! >>>>> >>>>> I encountered a similar case to what you've described in point #1. I used >>>>> a private plugin that writes some information to the metastorage. >>>>> After that I decided to get rid of that plugin, while the information had >>>>> already been written to the metastorage. >>>>> Following the approach that you described and implemented in the PR, I'll >>>>> need to work with the flag to ignore certain keys in the metastorage >>>>> forever. That's quite inconvenient. >>>>> Wouldn't it be better if we just limited the set of allowed types that >>>>> can be stored in the metastorage? Instead of a POJO, a Map will be >>>>> accepted. >>>>> >>>>> Denis >>>>> >>>>> On 18.12.2020, 13:59, "ткаленко кирилл" <tkalkir...@yandex.ru> wrote: >>>>> >>>>> Hello everybody! >>>>> >>>>> If you look at the stackTrace, the error is that deserialized objects >>>>> are being sent to listeners. >>>>> It may be more correct to send a raw arrays of bytes, and each plugin >>>>> will be able to process it if needed. >>>>> >>>>> 18.12.2020, 12:18, "Nikolay Izhikov" <nizhi...@apache.org>: >>>>>> Hello, Denis. >>>>>> >>>>>> It’s a known issue for me. >>>>>> Metastore is a private API, isn’t it? >>>>>> AFAICU it can occur for two reasons: >>>>>> >>>>>> * User migrates from custom Ignite fork that has private improvements or >>>>>> plugins that write to the metastore. >>>>>> * We have a blocker bug and just remove some internal class that can be >>>>>> written into metastore from distribution. >>>>>> >>>>>> I planned to fix it with some system flag. >>>>>> During startup administrator just sets a list of the metastore items >>>>>> that should be ignored. >>>>>> Please, take a look at the PR [1] >>>>>> >>>>>> WDYT? >>>>>> >>>>>>> it’s better to limit the metastorage with storing primitives only >>>>>> >>>>>> I think that ability to write object is very useful and should stay. >>>>>> >>>>>> [1] https://github.com/apache/ignite/pull/8221 >>>>>> >>>>>>> 18 дек. 2020 г., в 12:06, Mekhanikov Denis <dmekhani...@gmail.com> >>>>>>> написал(а): >>>>>>> >>>>>>> Hi everyone! >>>>>>> >>>>>>> Ignite has a limitation that it can’t work with custom classes put into >>>>>>> metastorage: https://issues.apache.org/jira/browse/IGNITE-13642 >>>>>>> If you put a POJO into the metastorage, then Ignite will try to >>>>>>> deserialize it using the classes it finds on the classpath. If it can’t >>>>>>> do the deserialization, then the node will fail. >>>>>>> >>>>>>> There is an opinion that the metastorage wasn’t design for a case when >>>>>>> classes that can disappear from Ignite distribution. >>>>>>> If we follow this path, then it’s better to limit the metastorage with >>>>>>> storing primitives only, so that it’s impossible to occasionally put >>>>>>> anything breaking. >>>>>>> If a piece of configuration is put into the metastorage by a plugin, >>>>>>> then the plugin will be in charge of deserializing the configuration, >>>>>>> and not Ignite. >>>>>>> >>>>>>> Alternatively we can try to fix the metastorage and make it ignore >>>>>>> deserialization errors when they occur. >>>>>>> >>>>>>> What do you think? >>>>>>> >>>>>>> Denis >
