Hi, I am using Spark 1.3.0, where I cannot load parquet files from more than one file system, say one s3n://... and another hdfs://..., which worked in older version, or if I set spark.sql.parquet.useDataSourceApi=false in 1.3.
One way to fix this is instead of get a single FileSystem from default configuration in ParquetRelation2, call Path.getFileSystem for each path. Here's the JIRA link and pull request: https://issues.apache.org/jira/browse/SPARK-6351 https://github.com/apache/spark/pull/5039 Thanks, -- Pei-Lun