JingsongLi commented on code in PR #9545:
URL: https://github.com/apache/paimon/pull/9545#discussion_r3986651844
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaEvolutionUtil.java:
##########
@@ -314,4 +317,16 @@ private static CastExecutor<InternalMap, InternalMap>
createMapCastExecutor(
CastedMap castedMap = CastedMap.from(castElementGetter);
return castedMap::replaceMap;
}
+
+ private static CastExecutor<InternalMap, InternalMap>
createMultisetCastExecutor(
+ MultisetType inputType, MultisetType targetType) {
+ CastElementGetter castElementGetter =
+ new CastElementGetter(
+
InternalArray.createElementGetter(inputType.getElementType()),
+ createCastExecutor(
+ inputType.getElementType(),
targetType.getElementType()));
+
+ CastedMap castedMap = CastedMap.fromKey(castElementGetter);
Review Comment:
[P1] Preserve multiplicities when evolved elements become equal
The new key-only lazy cast keeps one entry per old element even when an
allowed schema change makes their keys equal. I reproduced this with real table
writes followed by ALTER: `MULTISET<INT>` containing `{16777216:2, 16777217:3}`
can evolve to FLOAT, yielding two identical float keys; `InternalRowUtils.copy`
then overwrites the first count and returns multiplicity 3 instead of 5.
Dropping `a` from `MULTISET<ROW<a INT,b INT>>` containing `{(10,1):2,(20,1):3}`
has the same failure. INT to BIGINT is a passing control. These changes are
accepted by the current schema manager, so returning duplicate keys violates
the multiset representation and loses occurrences through normal
materialization. Either combine multiplicities for equal evolved elements, or
reject element evolutions that cannot preserve uniqueness before committing the
schema. Please add write/ALTER/read-and-copy regressions for these collisions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
