leesf commented on a change in pull request #4222:
URL: https://github.com/apache/hudi/pull/4222#discussion_r765382289
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -441,9 +441,10 @@ public void refreshTimeline() throws IOException {
return null;
}
+ jssc.setJobGroup(this.getClass().getSimpleName(), "Checking if input is
empty");
if ((!avroRDDOptional.isPresent()) || (avroRDDOptional.get().isEmpty())) {
LOG.info("No new data, perform empty commit.");
- return Pair.of(schemaProvider, Pair.of(checkpointStr, jssc.emptyRDD()));
+ return Pair.of(schemaProvider, Pair.of(checkpointStr, null));
Review comment:
> It saves one call to isEmpty() in `DeltaSync::writeToSink`, which
could further eliminate one spark job.
You would separate it in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
