Hi, Jingsong, I strongly agree that we should ensure state compatibility between different versions of paimon. State is a very important part of flink, such as sum / count / join and other calculations that require historical state. It is very important to ensure the compatibility of the state. If the state is incompatible, in simple cases we can discard the state and re-consume from the specified offset to overwrite the data. However, when paimon's merge-engine selects aggregation types such as sum / count, re-consumption cannot guarantee the correctness.
At the same time, paimon is currently in the process of rapid upgrade. Users are likely not to upgrade the flink, but only upgrade paimon. Therefore, we need to strongly ensure the compatibility of paimon. Best, Ming Li Jingsong Li <[email protected]> 于2024年5月27日周一 14:32写道： > Hi Paimon Dev, > > We have made many changes that do not guarantee the compatibility of > Flink job state, but in fact, we can guarantee it. We only need to > consider the design of VersionedSerializer when making modifications. > > I plan to introduce an annotation that reminds contributors to ensure > compatibility when making subsequent modifications to annotated > classes. > > For example, StateCompatibility annotation. > > What do you think? > > Best, > Jingsong >
