danny0405 commented on issue #19436: URL: https://github.com/apache/hudi/issues/19436#issuecomment-5186970282
Thanks for publishing the numbers. The allocation-free comparator makes this a much fairer comparison, and the result clearly shows that `String.compareTo` is faster—especially for long common prefixes. I do not think the pairwise comparator benchmark alone is enough to justify a format-dependent **persisted LSM ordering**, though. My main concern is cross-engine compatibility: - Flink's LSM bulk path uses `BinaryExternalSorter`. The input is serialized to `BinaryRowData`, and the generated string normalized key is based on `StringData.toBytes()`; the fallback comparison is therefore over the binary UTF-8 representation. - The Java LSM sorting helpers currently use the UTF-8 comparator as well. - Parquet defines the canonical `STRING` sort order as unsigned byte-wise comparison of its UTF-8 representation. Physical row order is not mandated by Parquet, but choosing UTF-16 would create a Hudi/JVM-specific ordering contract. That means a Parquet LSM run written by Flink can be UTF-8 ordered while the proposed Spark reader/writer expects UTF-16 order. The same physical table format would then have engine-dependent sorted-run semantics. Before locking this into the LSM format, could we add: 1. The JMH source, JVM flags, and operand-generation details so the microbenchmark is reproducible. Fixed pairwise operands can amplify branch-prediction and JDK-intrinsic effects. 2. A whole-sort benchmark (`Arrays.sort` and/or Spark `repartitionAndSortWithinPartitions`) with realistic key distributions, reporting wall time, CPU, allocation, shuffle, and spill. 3. A comparison against a precomputed UTF-8 sort representation—encode/materialize once per record, then compare bytes—rather than deriving UTF-8 order repeatedly inside every comparison. 4. A cross-engine test: Spark write → Flink read/merge and Flink write → Spark read/merge, using `U+E000` and `U+20000`. The reported worst case is a meaningful comparator-level difference (19.07 ns/op vs 3.76 ns/op), but we still need to establish the end-to-end write impact and weigh it against a permanent engine-specific table-format contract. If the end-to-end regression is material, I would prefer optimizing or caching the sort-key representation while retaining one canonical unsigned UTF-8 order for all LSM runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
