bhat-vinay commented on code in PR #10865:
URL: https://github.com/apache/hudi/pull/10865#discussion_r1535773675
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/S3EventsHoodieIncrSource.java:
##########
@@ -112,10 +110,15 @@ public S3EventsHoodieIncrSource(
QueryRunner queryRunner,
CloudDataFetcher cloudDataFetcher) {
super(props, sparkContext, sparkSession, schemaProvider);
+
+ if (getBooleanWithAltKeys(props, ENABLE_EXISTS_CHECK)) {
+ sparkSession.conf().set("spark.sql.files.ignoreMissingFiles", "true");
+ sparkSession.conf().set("spark.sql.files.ignoreCorruptFiles", "true");
Review Comment:
>yet the ROI might not be big enough to justify it
I am not sure of this, but since I have not benchmarked it, we do not know
if there are benefits.
>I think we should deprioritize this now and focus on other inflight PRs.
Yes, I already updated the associated jira yesterday and not looking into
this for the time being.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
