[
https://issues.apache.org/jira/browse/IOTDB-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071544#comment-17071544
]
WangChao commented on IOTDB-460:
--------------------------------
> How are you going to define the finishing of data pulling?
I think the new holder start a data group raft log after recevice partial slot
data. Slot pulling is finished when all slot data's log is committed. If
there are some members down, ether older or new, it's ok, raft will keep them
consistent.
> If so, adding nodes will be very... hard.
You can add a new command addNodes, it will add more than one node to the
cluster, then the meta will change once. And if only change once, it will be
ok.
I don't know how do you deal with such scene:
replica is 1, slot num is 12,
cluster starts with only A node, then add node B, so there six slots will
balance to B. Before B finish pulling slots from A, we add C, then there are
two slots need to be balanced from B to C. C will pulling slot from B, but B
has no such data. Or more complicated， if a slot balances from A to B, then
balances from B to A, and balances from A to B, then add C, C will get this
slot, how does it know B's slot data is new? If B is down in second step, C
will wait B to update it's slot data, or B does not know how to update it's
data, the newest partitiontable shows previous slot is him, how does it update
it's data?
I think this scene is too complicated to handle, I prefer that only when all
nodes's data is consistent with meta, we can do next meta change, this will
simple the design, and it does not has many constraints.
> [Distributed]Remove data of outdated slots
> ------------------------------------------
>
> Key: IOTDB-460
> URL: https://issues.apache.org/jira/browse/IOTDB-460
> Project: Apache IoTDB
> Issue Type: Improvement
> Reporter: Tian Jiang
> Priority: Major
> Labels: data-transfer, distributed
>
> In node addition/removal, the slots managed by a node will change. However,
> the data of corresponding slots cannot be removed yet because the new holders
> of the slots will pull such data to themselves. As the data pulling is issued
> by the new holders randomly, it is hard for the previous holders to find out
> when will the data be needless. As a result, the previous holder cannot
> delete the local data with confidence.
> It will be necessary to find a way for the previous holders to know when the
> data has been replicated to the new holders so that they will be able to
> remove the local data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
