cloud-fan commented on a change in pull request #27302: [SPARK-30506][SQL][DOC] Document for generic file source options/configs URL: https://github.com/apache/spark/pull/27302#discussion_r368838070
########## File path: docs/sql-data-sources-generic-options.md ########## @@ -0,0 +1,129 @@ +--- +layout: global +title: Generic File Source Options +displayTitle: Generic File Source Options +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +* Table of contents +{:toc} + +These generic options/configurations are effective only when using file-based sources such as Parquet, JSON and ORC. + +### Ignore Corrupt Files + +Spark allows you to use `spark.sql.files.ignoreCorruptFiles` to ignore corrupt files while reading data +from files. When set to true, the Spark jobs will continue to run when encountering corrupted files and +the contents that have been read will still be returned. + +To ignore corrupt files while reading data files, you can use: + +<div class="codetabs"> +<div data-lang="scala" markdown="1"> +{% include_example ignore_corrupt_files scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala %} +</div> + +<div data-lang="java" markdown="1"> +{% include_example ignore_corrupt_files java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java %} +</div> + +<div data-lang="python" markdown="1"> +{% include_example ignore_corrupt_files python/sql/datasource.py %} +</div> + +<div data-lang="r" markdown="1"> +{% include_example ignore_corrupt_files r/RSparkSQLExample.R %} +</div> +</div> + +### Ignore Missing Files + +Spark allows you to use `spark.sql.files.ignoreMissingFiles` to ignore missing files while reading data +from files. When set to true, the Spark jobs will continue to run when encountering missing files and +the contents that have been read will still be returned. + +To ignore missing files while reading data files, you can use: + +<div class="codetabs"> +<div data-lang="scala" markdown="1"> +{% include_example ignore_missing_files scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala %} +</div> + +<div data-lang="java" markdown="1"> +{% include_example ignore_missing_files java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java %} +</div> + +<div data-lang="python" markdown="1"> +{% include_example ignore_missing_files python/sql/datasource.py %} +</div> + +<div data-lang="r" markdown="1"> +{% include_example ignore_missing_files r/RSparkSQLExample.R %} + +</div> +</div> + +### Path Global Filter + +`pathGlobFilter` used to only include files with paths matching the pattern. +The syntax follows <code>org.apache.hadoop.fs.GlobFilter</code>. +It does not change the behavior of partition discovery. + +To load files with paths matching a given glob pattern while keeping the behavior of partition discovery, +you can use: + +<div class="codetabs"> +<div data-lang="scala" markdown="1"> +{% include_example load_with_path_glob_filter scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala %} +</div> + +<div data-lang="java" markdown="1"> +{% include_example load_with_path_glob_filter java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java %} +</div> + +<div data-lang="python" markdown="1"> +{% include_example load_with_path_glob_filter python/sql/datasource.py %} +</div> + +<div data-lang="r" markdown="1"> +{% include_example load_with_path_glob_filter r/RSparkSQLExample.R %} +</div> +</div> + +### Recursive File Lookup +`recursiveFileLookup` used to recursively load files and it disables partition inferring. Its default value is `false`. Review comment: ditto ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
