pratyakshsharma commented on a change in pull request #4925: URL: https://github.com/apache/hudi/pull/4925#discussion_r816193990
########## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java ########## @@ -379,23 +444,50 @@ private static String resetTarget(Config configuration, String database, String * Creates actual HoodieDeltaStreamer objects for every table/topic and does incremental sync. */ public void sync() { - for (TableExecutionContext context : tableExecutionContexts) { - try { - new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync(); - successTables.add(Helpers.getTableWithDatabase(context)); - } catch (Exception e) { - logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e); - failedTables.add(Helpers.getTableWithDatabase(context)); + if (tableExecutionContexts.isEmpty()) { + return; + } + if (tableExecutionContexts.get(0).getConfig().isFetchFromMultiSources) { + while (true) { Review comment: This essentially makes it an ever running job. I guess we do not want users to have resources up and running always doing ingestion with no option of having a one time job. From what I understand you are trying to introduce a continuous mode here, which is not recommended this way. Let us make it configurable? cc @nsivabalan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org