voonhous commented on code in PR #19493:
URL: https://github.com/apache/hudi/pull/19493#discussion_r3709520664
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkBinaryCopyClusteringAndValidationMeta.java:
##########
@@ -146,6 +149,18 @@ public void testAndValidateClusteringOutputFiles() throws
IOException {
client.commit(newCommitTime2, statuses2);
List<WriteStatus> statusList2 = statuses2.collect();
+ Set<String> commitTimeSet = new HashSet();
Review Comment:
Done, both are diamonds now. Checked the rest of the added lines too, these
were the only raw-type constructions.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkBinaryCopyClusteringAndValidationMeta.java:
##########
@@ -260,6 +268,115 @@ public void testSupportBinaryStreamCopy() throws
IOException {
Assertions.assertFalse(strategy.supportBinaryStreamCopy(groups, new
HashMap<>()));
}
+ @Test
+ public void testSupportStreamCopy() throws IOException {
+ HoodieWriteConfig writeConfig = new HoodieWriteConfig.Builder()
+ .withPath(basePath)
+ .withSchema(TRIP_EXAMPLE_SCHEMA)
+ .withEmbeddedTimelineServerEnabled(false)
+ .build();
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieTable table = HoodieSparkTable.create(writeConfig, engineContext,
metaClient);
+ SparkStreamCopyClusteringExecutionStrategy strategy =
+ new SparkStreamCopyClusteringExecutionStrategy(table, engineContext,
writeConfig);
+
+ MessageType legacySchema = new AvroSchemaConverter().convert(AVRO_SCHEMA);
+ Configuration conf = new Configuration();
+ conf.set("parquet.avro.write-old-list-structure", "false");
+ MessageType standardSchema = new
AvroSchemaConverter(conf).convert(AVRO_SCHEMA);
+
+ BloomFilter simpleBloomFilter = BloomFilterFactory.createBloomFilter(1000,
0.0001, 10000, SIMPLE.name());
+ BloomFilter dynmicBloomFilter = BloomFilterFactory.createBloomFilter(1000,
0.0001, 10000, DYNAMIC_V0.name());
+ // differs from the others in both bloom filter type code and list
structure
Review Comment:
Yes, typo. Renamed to `dynamicBloomFilter`. It was copy-pasted from
`testSupportBinaryStreamCopy`, which had the same misspelling, so I renamed
those four occurrences as well rather than leave two spellings in one file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
