[
https://issues.apache.org/jira/browse/IGNITE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815191#comment-17815191
]
Pavel Pereslegin edited comment on IGNITE-21230 at 2/7/24 9:32 AM:
-------------------------------------------------------------------
h4. Things about current implementation.
The idea of using {{IgniteDataTransferObject}} from AI2 didn't go well.
Mainly due to the need to make all objects {{Externalizable}}, in addition, our
entries have nested objects. For this to work properly, an intermediate layer
of "flat" {{Externalizable}} objects is needed.
As a result of various code shuffles and internal discussions, at the moment it
was decided to leave the simplest option possible. Each object is serialized by
its own external serializer. The current implementation of object serializers
is not aware of the data format version.
The version of the current format (1) is written at the very beginning (see the
description in the {{UpdateLogMarshallerImpl}} class).
h4. Ideas for the future data format:
It is necessary to maintain a registry of all types of serializable objects
(nested and external) and their serializers with a specific version.
When serializing, you must write current version of the object format
(serializer version).
When deserializing, you need to get the required serializer (object type +
version) from the registry.
>From the implementation side, it is supposed to add some "mix" of
>IgniteDataInput/IgniteDataOutput and ObjectInput/ObjectOutput which should be
>able to serialize/deserialize object types that are registered in the
>serializer registry.
was (Author: xtern):
h4. Things about current implementation.
The idea of using {{IgniteDataTransferObject}} from AI2 didn't go well.
Mainly due to the need to make all objects {{Externalizable}}, in addition, our
entries have nested objects. For this to work properly, an intermediate layer
of "flat" {{Externalizable}} objects is needed.
As a result of various code shuffles and internal discussions, at the moment it
was decided to leave the simplest option possible. Each object is serialized by
its own external serializer. The current implementation of object serializers
is not aware of the data format version.
The version of the current format (1) is written at the very beginning (see the
description in the {{UpdateLogMarshallerImpl}} class).
h4. Ideas for the future data format:
It is necessary to maintain a registry of all types of serializable objects
(nested and external) and their serializers with a specific version.
When serializing, you must write current version of the object format
(serializer version).
When deserializing, you need to get the required serializer (object type +
version) from the registry.
>From the implementation side, it is supposed to add some "mix" of
>IgniteDataInput/IgniteDataOutput and ObjectInput/ObjectOutput which should be
>able to serialize/deserialize object types that are registered in the
>serializer registry.
> Improve serialization of catalog commands
> -----------------------------------------
>
> Key: IGNITE-21230
> URL: https://issues.apache.org/jira/browse/IGNITE-21230
> Project: Ignite
> Issue Type: Improvement
> Reporter: Konstantin Orlov
> Assignee: Pavel Pereslegin
> Priority: Major
> Labels: ignite-3
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, catalog uses java's built-in serialisation in order to persists
> descriptors and update entries. Although it is most simple way to address
> conversion of POJO to byte array, the built-in serialisation is not efficient
> in terms of occupied space. Given that catalog may contain thousands and
> thousands of objects, it may result in increased load on a metastorage.
> To address this issue, let's introduce custom serialisation format for
> catalog's descriptors and update entries.
> NB: we should keep in mind that descriptors may evolve over time, thus we
> should consider support of versioning for descriptors
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
