Blazer-007 commented on code in PR #4044:
URL: https://github.com/apache/gobblin/pull/4044#discussion_r1874733727
##########
gobblin-core/src/main/java/org/apache/gobblin/source/DatePartitionedNestedRetriever.java:
##########
@@ -129,8 +134,15 @@ public List<FileInfo> getFilesToProcess(long minWatermark,
int maxFilesToReturn)
new FileInfo(fileStatus.getPath().toString(),
fileStatus.getLen(), date.getMillis(), partitionPath));
}
}
+
+ if (growthTracker.isAnotherMilestone(iteration++)) {
+ LOG.info("~{}~ collected files to process", filesToProcess.size());
Review Comment:
Updated the log lines , it was confusing initially.
filesToProcess -- is the list of file we are collecting here,
only goal here to add log inside is to find if code is stucked or not as we
saw that due to too much granularity code was stucked and we have to get thread
dump to see where the code was stucked and nothing was in log lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
