Jackie-Jiang commented on code in PR #20113:
URL: https://github.com/apache/pinot/pull/20113#discussion_r4532008596


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CrcUtils.java:
##########
@@ -42,30 +43,58 @@ private CrcUtils() {
   private static final Logger LOGGER = LoggerFactory.getLogger(CrcUtils.class);
   private static final int BUFFER_SIZE = 65536;
   private static final String CRC_FILE_EXTENSTION = ".crc";
-  private static final List<String> DATA_FILE_EXTENSIONS = 
Arrays.asList(".fwd", ".dict");
+  private static final List<String> DATA_FILE_EXTENSIONS_V1 = 
Arrays.asList(".fwd", ".dict");
+  // Null vectors count as data in version 2, because a null is stored as the 
default value in the forward index
+  private static final List<String> DATA_FILE_EXTENSIONS_V2 =
+      Arrays.asList(".fwd", ".dict", 
V1Constants.Indexes.NULLVALUE_VECTOR_FILE_EXTENSION);
 
+  private static volatile int _defaultDataCrcVersion = 
V1Constants.DataCrc.VERSION_1;
+
+  /// Sets the data CRC version used for segments built in this JVM.
+  public static void setDefaultDataCrcVersion(int dataCrcVersion) {
+    _defaultDataCrcVersion = validateDataCrcVersion(dataCrcVersion);
+  }
+
+  public static int getDefaultDataCrcVersion() {
+    return _defaultDataCrcVersion;
+  }
 
   public static long computeCrc(File indexDir) throws IOException {
     List<File> allNormalFiles = new ArrayList<>();
-    collectFiles(indexDir, allNormalFiles, false);
+    collectFiles(indexDir, allNormalFiles, null);
     Collections.sort(allNormalFiles);
     return crcForFiles(allNormalFiles);
   }
 
   public static long computeDataCrc(File indexDir) throws IOException {
+    return computeDataCrc(indexDir, V1Constants.DataCrc.VERSION_1);
+  }
+
+  /// Computes the data CRC with the given version. Version 2 equals version 1 
when the segment has no null vectors.
+  public static long computeDataCrc(File indexDir, int dataCrcVersion) throws 
IOException {
+    List<String> dataFileExtensions =

Review Comment:
   Consider encoding the version in the existing data CRC long. Adler32 uses 
only the lower 32 bits, so we can make version 0 the default/legacy algorithm 
and version 1 the algorithm including null vectors:
   
   ```java
   return ((long) version << 32) | checksum;
   ```
   
   Version 0 preserves every existing checksum value. Version 1 remains 
positive and cannot match a legacy checksum. The existing long equality checks 
then compare both version and checksum automatically, including on old servers. 
The OSS metadata, ZK, upload-response, and valid-doc-ID paths I checked 
preserve the full long value.
   
   This would remove the appended integer in creation.meta, the separate 
metadata field/SPI getter/JSON property, and the need to propagate a separate 
version through ZK and upload responses. Keep supported-version validation and 
leave negative unavailable sentinels unencoded.
   
   It also closes the mixed-version gap in the current implementation: a legacy 
copy containing NULL can have the same data CRC as a new-algorithm copy 
containing the corresponding stored default. The separate version currently 
does not participate in hasSameCRC, so the claim that these cases always fall 
back to full CRC is not correct.
   
   The deliberate tradeoff is that cross-version data CRCs also differ for 
segments without nulls. Full CRC equality should remain sufficient; otherwise 
the replica downloads the committed copy. Please cover the encoding and 
round-trip values above 0xffffffff, plus mixed-version comparisons with equal 
and unequal full CRCs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to