Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
vinothchandar merged PR #10492: URL: https://github.com/apache/hudi/pull/10492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
vinothchandar commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1896326101 CI passes. Can merge once @jonvex confirms the cases above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
vinothchandar commented on code in PR #10492:
URL: https://github.com/apache/hudi/pull/10492#discussion_r1454069459
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamerSchemaEvolutionQuick.java:
##
@@ -59,25 +59,34 @@ public void teardown() throws Exception {
}
protected static Stream testArgs() {
+boolean fullTest = false;
Stream.Builder b = Stream.builder();
-//only testing row-writer enabled for now
-for (Boolean rowWriterEnable : new Boolean[] {true}) {
- for (Boolean nullForDeletedCols : new Boolean[] {false, true}) {
-for (Boolean useKafkaSource : new Boolean[] {false, true}) {
- for (Boolean addFilegroups : new Boolean[] {false, true}) {
-for (Boolean multiLogFiles : new Boolean[] {false, true}) {
- for (Boolean shouldCluster : new Boolean[] {false, true}) {
-for (String tableType : new String[] {"COPY_ON_WRITE",
"MERGE_ON_READ"}) {
- if (!multiLogFiles || tableType.equals("MERGE_ON_READ")) {
-b.add(Arguments.of(tableType, shouldCluster, false,
rowWriterEnable, addFilegroups, multiLogFiles, useKafkaSource,
nullForDeletedCols));
+if (fullTest) {
+ //only testing row-writer enabled for now
+ for (Boolean rowWriterEnable : new Boolean[] {true}) {
+for (Boolean nullForDeletedCols : new Boolean[] {false, true}) {
+ for (Boolean useKafkaSource : new Boolean[] {false, true}) {
+for (Boolean addFilegroups : new Boolean[] {false, true}) {
+ for (Boolean multiLogFiles : new Boolean[] {false, true}) {
+for (Boolean shouldCluster : new Boolean[] {false, true}) {
+ for (String tableType : new String[] {"COPY_ON_WRITE",
"MERGE_ON_READ"}) {
+if (!multiLogFiles || tableType.equals("MERGE_ON_READ")) {
+ b.add(Arguments.of(tableType, shouldCluster, false,
rowWriterEnable, addFilegroups, multiLogFiles, useKafkaSource,
nullForDeletedCols));
+}
}
}
+b.add(Arguments.of("MERGE_ON_READ", false, true,
rowWriterEnable, addFilegroups, multiLogFiles, useKafkaSource,
nullForDeletedCols));
}
- b.add(Arguments.of("MERGE_ON_READ", false, true,
rowWriterEnable, addFilegroups, multiLogFiles, useKafkaSource,
nullForDeletedCols));
}
}
}
}
+} else {
Review Comment:
```
String tableType = COW, MOR
Boolean shouldCluster = true
Boolean shouldCompact = true
Boolean rowWriterEnable = true
Boolean addFilegroups = true
Boolean multiLogFiles = true
Boolean useKafkaSource= false, true
Boolean allowNullForDeletedCols=false,true
```
I wonder if we just do sth like this. with new
file groups, multiple log files, alongside cluster and compaction, should be
the more complex (superset) scenario. no?
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamerSchemaEvolutionQuick.java:
##
@@ -97,19 +106,27 @@ protected static Stream testReorderedColumn() {
}
protected static Stream testParamsWithSchemaTransformer() {
+boolean fullTest = false;
Stream.Builder b = Stream.builder();
-for (Boolean useTransformer : new Boolean[] {false, true}) {
- for (Boolean setSchema : new Boolean[] {false, true}) {
-for (Boolean rowWriterEnable : new Boolean[] {true}) {
- for (Boolean nullForDeletedCols : new Boolean[] {false, true}) {
-for (Boolean useKafkaSource : new Boolean[] {false, true}) {
- for (String tableType : new String[] {"COPY_ON_WRITE",
"MERGE_ON_READ"}) {
-b.add(Arguments.of(tableType, rowWriterEnable, useKafkaSource,
nullForDeletedCols, useTransformer, setSchema));
+if (fullTest) {
+ for (Boolean useTransformer : new Boolean[] {false, true}) {
+for (Boolean setSchema : new Boolean[] {false, true}) {
+ for (Boolean rowWriterEnable : new Boolean[] {true}) {
+for (Boolean nullForDeletedCols : new Boolean[] {false, true}) {
+ for (Boolean useKafkaSource : new Boolean[] {false, true}) {
+for (String tableType : new String[] {"COPY_ON_WRITE",
"MERGE_ON_READ"}) {
+ b.add(Arguments.of(tableType, rowWriterEnable,
useKafkaSource, nullForDeletedCols, useTransformer, setSchema));
+}
}
}
}
}
}
+} else
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
linliu-code commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1894464626 @jonvex, when is "fullTest" set to "true"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
hudi-bot commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1888274766 ## CI report: * 1b5d4ba50a611488bdc533914c88475ced19fd99 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21938) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
hudi-bot commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1888205602 ## CI report: * c262717fa9b3158690de5f6030c84ae6262b9c74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21935) * 1b5d4ba50a611488bdc533914c88475ced19fd99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21938) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
hudi-bot commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1888092990 ## CI report: * c262717fa9b3158690de5f6030c84ae6262b9c74 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21935) * 1b5d4ba50a611488bdc533914c88475ced19fd99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21938) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
hudi-bot commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1888085542 ## CI report: * c262717fa9b3158690de5f6030c84ae6262b9c74 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21935) * 1b5d4ba50a611488bdc533914c88475ced19fd99 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
hudi-bot commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1888004714 ## CI report: * c262717fa9b3158690de5f6030c84ae6262b9c74 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21935) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7296] Reduce CI Time by Minimizing Duplicate Code Coverage in Tests [hudi]
hudi-bot commented on PR #10492: URL: https://github.com/apache/hudi/pull/10492#issuecomment-1887943396 ## CI report: * c262717fa9b3158690de5f6030c84ae6262b9c74 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
