wgtmac commented on code in PR #3827:
URL: https://github.com/apache/parquet-java/pull/3827#discussion_r4240486485
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestParquetWriter.java:
##########
@@ -335,6 +337,114 @@ public void testParquetFileWithBloomFilter() throws
IOException {
}
}
+ @Test
+ public void testBloomFiltersKeepCollidingDotStringPathsDistinct() throws
IOException {
+ Map<ColumnPath, BloomFilter> bloomFilters =
writeBloomFiltersForCollidingPaths();
+ BloomFilter topLevelBloom = bloomFilters.get(ColumnPath.get("a.b"));
+ BloomFilter nestedBloom = bloomFilters.get(ColumnPath.get("a", "b"));
+
+ for (int valueId = 0; valueId < 10; valueId++) {
+ long topLevelHash = LongHashFunction.xx(0)
+ .hashBytes(Binary.fromString("top-" + valueId).toByteBuffer());
+ long nestedHash = LongHashFunction.xx(0)
+ .hashBytes(Binary.fromString("nested-" + valueId).toByteBuffer());
+ assertThat(topLevelBloom.findHash(topLevelHash)).isTrue();
+ assertThat(nestedBloom.findHash(nestedHash)).isTrue();
+ }
+ }
+
+ @Test
+ public void testBloomFilterPathCollisionGolden() throws IOException {
+ Map<ColumnPath, BloomFilter> bloomFilters =
writeBloomFiltersForCollidingPaths();
+ String actual = String.format(
+ "{\n"
+ + " \"topLevelPath\" : [ \"a.b\" ],\n"
+ + " \"topLevelContainsOwnBloomValues\" : %s,\n"
+ + " \"nestedPath\" : [ \"a\", \"b\" ],\n"
+ + " \"nestedContainsOwnBloomValues\" : %s\n"
+ + "}\n",
+ containsAllBloomValues(bloomFilters.get(ColumnPath.get("a.b")),
"top-"),
+ containsAllBloomValues(bloomFilters.get(ColumnPath.get("a", "b")),
"nested-"));
+
+ try (InputStream input =
TestParquetWriter.class.getResourceAsStream("/bloom-filter-path-collision.json"))
{
Review Comment:
The expected file repeats the values built two lines above, so there are two
copies of the same expectations. Let's remove this resource file to make it
easier to maintain.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java:
##########
@@ -1078,6 +1078,16 @@ public void writeDataPage(
* @param bloomFilter the bloom filter of column values
*/
public void addBloomFilter(String column, BloomFilter bloomFilter) {
+ addBloomFilterForPath(ColumnPath.fromDotString(column), bloomFilter);
Review Comment:
`fromDotString` splits on dots. A caller who passes a name that contains a
dot (for example the top-level `a.b` column this PR fixes) now builds a path
that does not match the column, so their Bloom filter is dropped without any
error or log. The javadoc above still says "the column name".
Perhaps we need to deprecate this method and add a comment to point callers
at `addBloomFilterForPath`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
