[
https://issues.apache.org/jira/browse/SPARK-43269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neil Jonkers updated SPARK-43269:
---------------------------------
Description:
Hello,
{{With *spark.sql.files.ignoreMissingFiles=true* we notice
[readParquetFootersInParallel |#L438]can still encounter
{*}FileNotFoundException{*}. }}
I notice function readParquetFootersInParallel handle the scenario where
{{{*}spark.sql.files.ignoreCorruptFiles=true{*}.}}
My question: Would it be feasible to support the scenario where
*spark.sql.files.ignoreMissingFiles=true* in the function:
readParquetFootersInParallel as well ? To prevent application failure due to
{{*FileNotFoundException.*}}
Thank you
was:
Hello,
{{With }}*spark.sql.files.ignoreMissingFiles=true*{{ we notice
[readParquetFootersInParallel |#L438]can still encounter
{*}FileNotFoundException{*}. }}
I notice function readParquetFootersInParallel handle the scenario where
{{{*}spark.sql.files.ignoreCorruptFiles=true{*}.}}
My question: Would it be feasible to support the scenario where
*spark.sql.files.ignoreMissingFiles=true* in the function:
readParquetFootersInParallel as well ? To prevent application failure due to
{{*FileNotFoundException.*}}
Thank you
> Adding support for MissingFiles when spark.sql.parquet.mergeSchema=true
> -----------------------------------------------------------------------
>
> Key: SPARK-43269
> URL: https://issues.apache.org/jira/browse/SPARK-43269
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Neil Jonkers
> Priority: Minor
>
> Hello,
> {{With *spark.sql.files.ignoreMissingFiles=true* we notice
> [readParquetFootersInParallel |#L438]can still encounter
> {*}FileNotFoundException{*}. }}
> I notice function readParquetFootersInParallel handle the scenario where
> {{{*}spark.sql.files.ignoreCorruptFiles=true{*}.}}
> My question: Would it be feasible to support the scenario where
> *spark.sql.files.ignoreMissingFiles=true* in the function:
> readParquetFootersInParallel as well ? To prevent application failure due to
> {{*FileNotFoundException.*}}
>
> Thank you
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
