Sergey, In my understanding the only difficult thing here is how to disallow usage of not-yet-confirmed metadata during serialization. As we assume metadata updates to be relatively rare, can we always initiate message send from coordinator node? In this case we will have total order of all metadata updates, what will make conflict resolution trivial.
Alex, how do you think - will this approach work? I am concerned about coordinator failover. On Thu, Jan 12, 2017 at 4:09 PM, Sergey Chugunov <sergey.chugu...@gmail.com> wrote: > Hello Ignite devs, > > I came up with the following design proposal for IGNITE-4302 > <https://issues.apache.org/jira/browse/IGNITE-4302>, which is very similar > to described here > <http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4157-design- > proposal-td11996.html> > proposal for IGNITE-4157 <https://issues.apache.org/ > jira/browse/IGNITE-4157>, > "*Use discovery custom messages instead of marshaller cache*". > > Protocol is very similar to IGNITE-4157 and based on exchanging two > messages: *MetadataProposed*/*MetadataAccepted*. > > 1. When a node wants to add or update existing metadata, it sends > *MetadataProposed* message with the update. > 2. Coordinator node checks for any conflicts with existing metadata for > this typeId. In case of no conflicts coordinator assigns a version > number > to new metadata and sends Proposed message further. > In case of conflict it simply marks this message as IN_CONFLICT and > sends it to the original node. > 3. All nodes upon receiving Proposed message update local metadata > including version number and put it to "pending_acceptance" status. [1] > 4. Coordinator acknowledges proposed message with *MetadataAccepted* > message, marking it with the version number from proposed message. > 5. Each node receiving accepted message checks local version number and > the one from the message. If they are equal, metadata is considered as > accepted by the cluster. [2] > > New nodes joining the cluster receive metadata on discovery phase. > > As discovery messages are delivered to clients asynchronously, there is a > possibility that cluster may consider and agree upon update to metadata > before client receives initial proposed message. > In that case I suggest to attach version number of metadata BinaryObject > was serialized with to that BinaryObject so client may check whether it can > deserialize this object safely or need to request some updated version of > metadata. > > [1] When a metadata is marked as "pending_acceptance" it should be > prohibited to use it to serialize binary objects but it still can be used > for deserialization. > At the same time from code perspective I didn't find a clear distinction > like "this method is used only for serialization and that method is called > only during deserialization". So for now the only way I see is to block all > operations for a given typeId until MetadataAccepted message for this > typeId is arrived. > > [2] If MetadataAccepted message has a version number less than currently > pending, it is ignored and all threads waiting for this metadata to be > accepted remain blocked. > It may be possible for new joining node to receive a MetadataAccepted > message with version number greater than local copy, in that case it is > fine to apply metadata update and declare metadata accepted. > > > Please share with me your thoughts about suggested design. Any > improvements, missed corner cases or other drawbacks are really > appreciated. > > Thanks, > Sergey. >
