Maxim Gekk created SPARK-24836: ---------------------------------- Summary: New option - ignoreExtension Key: SPARK-24836 URL: https://issues.apache.org/jira/browse/SPARK-24836 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 2.3.1 Reporter: Maxim Gekk
Need to add new option for Avro datasource - *ignoreExtension*. It should control ignoring of the .avro extensions. If it is set to *true* (by default), files with and without .avro extensions should be loaded. Example of usage: {code:scala} spark .read .option("ignoreExtension", false) .avro("path to avro files") {code} The option duplicates Hadoop's config avro.mapred.ignore.inputs.without.extension which is taken into account by Avro datasource now and can be set like: {code:scala} spark .sqlContext .sparkContext .hadoopConfiguration .set("avro.mapred.ignore.inputs.without.extension", "true") {code} The ignoreExtension option must override avro.mapred.ignore.inputs.without.extension. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org