Re: IGNITE-4302 discovery messages to exchange binary metadata - design proposal

[Sergey Chugunov]

Vladimir, Alexey,

Let me describe my current vision of the task; please correct me if I'm
wrong somewhere.


   1. The only class adding and retrieving metadata to/from sys-cache is
   *CacheObjectBinaryProcessorImpl*.

   I expect all code managing discovery events flow to be placed here.

   Two the most important methods here are *addMeta(...)* and
   *metadata(...)*.
   I found out that both methods may be called during serialization and
   deserialization phases.

   *So at the moment I'm going to block all threads trying to do anything
   with not-yet-confirmed metadata*. I don't see any possibility to
   distinguish serialize call from deserialize call on BinaryProcessor level.
   If you know how to implement this more efficiently, please let me know.

   2. I identified two use cases where I need to manage versioning of
   metadata.

   The first one is related to situation when cluster accepts new version
   of metadata and some client node still has older version of metadata.
   In that case client needs a version number on incoming binary object to
   compare it with version number of locally available metadata for this
   object and to request metadata if needed.

   The second case occurs when two changes to the same typeId are proposed
   by different nodes in cluster.
   In this situation it is possible that one change is already spread
   across the cluster when another one is still in progress. Acknowledge
   message for the first change should not unblock any threads waiting for the
   second one.
   The only way I see to achieve this is to add version number as described
   in initial email.

   I need some assistance here to think over a change of adding metadata
   version number to current binary object format.


Thanks,
Sergey.



On Fri, Jan 13, 2017 at 12:28 PM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Vladimir,
>
> Given that metadata updates will be initiated with DiscoveryCustomEvent, I
> see no problems neither with total message ordering (the order will be the
> same on all nodes and consistent with discovery events) nor with failover -
> the event is marked with @TcpDiscoveryEnsureDelivery, so the event should
> be fired even in the case of coordinator failure.
>
> I think handling metadata on clients might be tricky, however fore-fetch
> approach is proven to work in the scope of IGNITE-4157.
>
> 2017-01-13 12:04 GMT+03:00 Vladimir Ozerov :
>
> > Sergey,
> >
> > In my understanding the only difficult thing here is how to disallow
> usage
> > of not-yet-confirmed metadata during serialization. As we assume metadata
> > updates to be relatively rare, can we always initiate message send from
> > coordinator node? In this case we will have total order of all metadata
> > updates, what will make conflict resolution trivial.
> >
> > Alex, how do you think - will this approach work? I am concerned about
> > coordinator failover.
> >
> > On Thu, Jan 12, 2017 at 4:09 PM, Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Hello Ignite devs,
> > >
> > > I came up with the following design proposal for IGNITE-4302
> > > , which is very
> > similar
> > > to described here
> > >  > com/IGNITE-4157-design-
> > > proposal-td11996.html>
> > > proposal for IGNITE-4157  > > jira/browse/IGNITE-4157>,
> > > "*Use discovery custom messages instead of marshaller cache*".
> > >
> > > Protocol is very similar to IGNITE-4157 and based on exchanging two
> > > messages: *MetadataProposed*/*MetadataAccepted*.
> > >
> > >1. When a node wants to add or update existing metadata, it sends
> > >*MetadataProposed* message with the update.
> > >2. Coordinator node checks for any conflicts with existing metadata
> > for
> > >this typeId. In case of no conflicts coordinator assigns a version
> > > number
> > >to new metadata and sends Proposed message further.
> > >In case of conflict it simply marks this message as IN_CONFLICT and
> > >sends it to the original node.
> > >3. All nodes upon receiving Proposed message update local metadata
> > >including version number and put it to "pending_acceptance" status.
> > [1]
> > >4. Coordinator acknowledges proposed message with *MetadataAccepted*
> > >message, marking it with the version number from proposed message.
> > >5. Each node receiving accepted message checks local version number
> > and
> > >the one from the message. If they are equal, metadata is considered
> as
> > >accepted by the cluster. [2]
> > >
> > > New nodes joining the cluster receive metadata on discovery phase.
> > >
> > > As discovery messages are delivered to clients asynchronously, there
> is a
> > > possibility that cluster may consider and agree upon update to metadata
> > > before client receives initial proposed message.
> > > In that case I suggest to attach version n

Re: IGNITE-4302 discovery messages to exchange binary metadata - design proposal

[Alexey Goncharuk]

Vladimir,

Given that metadata updates will be initiated with DiscoveryCustomEvent, I
see no problems neither with total message ordering (the order will be the
same on all nodes and consistent with discovery events) nor with failover -
the event is marked with @TcpDiscoveryEnsureDelivery, so the event should
be fired even in the case of coordinator failure.

I think handling metadata on clients might be tricky, however fore-fetch
approach is proven to work in the scope of IGNITE-4157.

2017-01-13 12:04 GMT+03:00 Vladimir Ozerov :

> Sergey,
>
> In my understanding the only difficult thing here is how to disallow usage
> of not-yet-confirmed metadata during serialization. As we assume metadata
> updates to be relatively rare, can we always initiate message send from
> coordinator node? In this case we will have total order of all metadata
> updates, what will make conflict resolution trivial.
>
> Alex, how do you think - will this approach work? I am concerned about
> coordinator failover.
>
> On Thu, Jan 12, 2017 at 4:09 PM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > Hello Ignite devs,
> >
> > I came up with the following design proposal for IGNITE-4302
> > , which is very
> similar
> > to described here
> >  com/IGNITE-4157-design-
> > proposal-td11996.html>
> > proposal for IGNITE-4157  > jira/browse/IGNITE-4157>,
> > "*Use discovery custom messages instead of marshaller cache*".
> >
> > Protocol is very similar to IGNITE-4157 and based on exchanging two
> > messages: *MetadataProposed*/*MetadataAccepted*.
> >
> >1. When a node wants to add or update existing metadata, it sends
> >*MetadataProposed* message with the update.
> >2. Coordinator node checks for any conflicts with existing metadata
> for
> >this typeId. In case of no conflicts coordinator assigns a version
> > number
> >to new metadata and sends Proposed message further.
> >In case of conflict it simply marks this message as IN_CONFLICT and
> >sends it to the original node.
> >3. All nodes upon receiving Proposed message update local metadata
> >including version number and put it to "pending_acceptance" status.
> [1]
> >4. Coordinator acknowledges proposed message with *MetadataAccepted*
> >message, marking it with the version number from proposed message.
> >5. Each node receiving accepted message checks local version number
> and
> >the one from the message. If they are equal, metadata is considered as
> >accepted by the cluster. [2]
> >
> > New nodes joining the cluster receive metadata on discovery phase.
> >
> > As discovery messages are delivered to clients asynchronously, there is a
> > possibility that cluster may consider and agree upon update to metadata
> > before client receives initial proposed message.
> > In that case I suggest to attach version number of metadata BinaryObject
> > was serialized with to that BinaryObject so client may check whether it
> can
> > deserialize this object safely or need to request some updated version of
> > metadata.
> >
> > [1] When a metadata is marked as "pending_acceptance" it should be
> > prohibited to use it to serialize binary objects but it still can be used
> > for deserialization.
> > At the same time from code perspective I didn't find a clear distinction
> > like "this method is used only for serialization and that method is
> called
> > only during deserialization". So for now the only way I see is to block
> all
> > operations for a given typeId until MetadataAccepted message for this
> > typeId is arrived.
> >
> > [2] If MetadataAccepted message has a version number less than currently
> > pending, it is ignored and all threads waiting for this metadata to be
> > accepted remain blocked.
> > It may be possible for new joining node to receive a MetadataAccepted
> > message with version number greater than local copy, in that case it is
> > fine to apply metadata update and declare metadata accepted.
> >
> >
> > Please share with me your thoughts about suggested design. Any
> > improvements, missed corner cases or other drawbacks are really
> > appreciated.
> >
> > Thanks,
> > Sergey.
> >
>

Re: IGNITE-4302 discovery messages to exchange binary metadata - design proposal

[Vladimir Ozerov]

Sergey,

In my understanding the only difficult thing here is how to disallow usage
of not-yet-confirmed metadata during serialization. As we assume metadata
updates to be relatively rare, can we always initiate message send from
coordinator node? In this case we will have total order of all metadata
updates, what will make conflict resolution trivial.

Alex, how do you think - will this approach work? I am concerned about
coordinator failover.

On Thu, Jan 12, 2017 at 4:09 PM, Sergey Chugunov 
wrote:

> Hello Ignite devs,
>
> I came up with the following design proposal for IGNITE-4302
> , which is very similar
> to described here
>  proposal-td11996.html>
> proposal for IGNITE-4157  jira/browse/IGNITE-4157>,
> "*Use discovery custom messages instead of marshaller cache*".
>
> Protocol is very similar to IGNITE-4157 and based on exchanging two
> messages: *MetadataProposed*/*MetadataAccepted*.
>
>1. When a node wants to add or update existing metadata, it sends
>*MetadataProposed* message with the update.
>2. Coordinator node checks for any conflicts with existing metadata for
>this typeId. In case of no conflicts coordinator assigns a version
> number
>to new metadata and sends Proposed message further.
>In case of conflict it simply marks this message as IN_CONFLICT and
>sends it to the original node.
>3. All nodes upon receiving Proposed message update local metadata
>including version number and put it to "pending_acceptance" status. [1]
>4. Coordinator acknowledges proposed message with *MetadataAccepted*
>message, marking it with the version number from proposed message.
>5. Each node receiving accepted message checks local version number and
>the one from the message. If they are equal, metadata is considered as
>accepted by the cluster. [2]
>
> New nodes joining the cluster receive metadata on discovery phase.
>
> As discovery messages are delivered to clients asynchronously, there is a
> possibility that cluster may consider and agree upon update to metadata
> before client receives initial proposed message.
> In that case I suggest to attach version number of metadata BinaryObject
> was serialized with to that BinaryObject so client may check whether it can
> deserialize this object safely or need to request some updated version of
> metadata.
>
> [1] When a metadata is marked as "pending_acceptance" it should be
> prohibited to use it to serialize binary objects but it still can be used
> for deserialization.
> At the same time from code perspective I didn't find a clear distinction
> like "this method is used only for serialization and that method is called
> only during deserialization". So for now the only way I see is to block all
> operations for a given typeId until MetadataAccepted message for this
> typeId is arrived.
>
> [2] If MetadataAccepted message has a version number less than currently
> pending, it is ignored and all threads waiting for this metadata to be
> accepted remain blocked.
> It may be possible for new joining node to receive a MetadataAccepted
> message with version number greater than local copy, in that case it is
> fine to apply metadata update and declare metadata accepted.
>
>
> Please share with me your thoughts about suggested design. Any
> improvements, missed corner cases or other drawbacks are really
> appreciated.
>
> Thanks,
> Sergey.
>

IGNITE-4302 discovery messages to exchange binary metadata - design proposal

[Sergey Chugunov]

Hello Ignite devs,

I came up with the following design proposal for IGNITE-4302
, which is very similar
to described here

proposal for IGNITE-4157 ,
"*Use discovery custom messages instead of marshaller cache*".

Protocol is very similar to IGNITE-4157 and based on exchanging two
messages: *MetadataProposed*/*MetadataAccepted*.

   1. When a node wants to add or update existing metadata, it sends
   *MetadataProposed* message with the update.
   2. Coordinator node checks for any conflicts with existing metadata for
   this typeId. In case of no conflicts coordinator assigns a version number
   to new metadata and sends Proposed message further.
   In case of conflict it simply marks this message as IN_CONFLICT and
   sends it to the original node.
   3. All nodes upon receiving Proposed message update local metadata
   including version number and put it to "pending_acceptance" status. [1]
   4. Coordinator acknowledges proposed message with *MetadataAccepted*
   message, marking it with the version number from proposed message.
   5. Each node receiving accepted message checks local version number and
   the one from the message. If they are equal, metadata is considered as
   accepted by the cluster. [2]

New nodes joining the cluster receive metadata on discovery phase.

As discovery messages are delivered to clients asynchronously, there is a
possibility that cluster may consider and agree upon update to metadata
before client receives initial proposed message.
In that case I suggest to attach version number of metadata BinaryObject
was serialized with to that BinaryObject so client may check whether it can
deserialize this object safely or need to request some updated version of
metadata.

[1] When a metadata is marked as "pending_acceptance" it should be
prohibited to use it to serialize binary objects but it still can be used
for deserialization.
At the same time from code perspective I didn't find a clear distinction
like "this method is used only for serialization and that method is called
only during deserialization". So for now the only way I see is to block all
operations for a given typeId until MetadataAccepted message for this
typeId is arrived.

[2] If MetadataAccepted message has a version number less than currently
pending, it is ignored and all threads waiting for this metadata to be
accepted remain blocked.
It may be possible for new joining node to receive a MetadataAccepted
message with version number greater than local copy, in that case it is
fine to apply metadata update and declare metadata accepted.


Please share with me your thoughts about suggested design. Any
improvements, missed corner cases or other drawbacks are really appreciated.

Thanks,
Sergey.