GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/15813
[SPARK-18362][SQL] Use TextFileFormat in JsonFileFormat and CSVFileFormat
## What changes were proposed in this pull request?
This patch significantly improves the IO / file listing performance of
schema inference in Spark's built-in CSV and JSON data sources.
Previously, these data sources used the legacy `SparkContext.hadoopFile`
and `SparkContext.hadoopRDD` methods to read files during their schema
inference steps, causing huge file-listing bottlenecks on the driver.
This patch refactors this logic to use Spark SQL's `text` data source to
read files during this step. The text data source still performs some
unnecessary file listing (since in theory we already have resolved the table
prior to schema inference and therefore should be able to scan without
performing _any_ extra listing), but that listing is much faster and takes
place in parallel. In one production workload operating over tens of thousands
of JSON files, this change managed to reduce schema inference time from 7
minutes to 2 minutes.
## How was this patch tested?
Existing unit tests, plus manual benchmarking on a production workload.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
use-text-data-source-in-csv-and-json
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15813.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15813
----
commit b697436d2e5eefb8007cbe8add57426f40668923
Author: Josh Rosen <[email protected]>
Date: 2016-11-03T23:53:39Z
Use text data source in CSV and JSON data sources.
commit cfb2f413023d49bfd655012717a99f3dc05702dd
Author: Josh Rosen <[email protected]>
Date: 2016-11-04T00:40:37Z
Don't check file existence.
commit 0fda0ecf01b74c3451429dce8943fc73eb2f7840
Author: Josh Rosen <[email protected]>
Date: 2016-11-04T01:10:38Z
Fix name.
commit eb8ddfd7c80c349edd20ce64f7dc69b2b75f022f
Author: Josh Rosen <[email protected]>
Date: 2016-11-04T18:33:01Z
Actually use `lines`
commit acce60d4f584f549a44f6dd90e3db5e6694a61f0
Author: Josh Rosen <[email protected]>
Date: 2016-11-04T22:53:00Z
Clean up CSV code to reduce number of scans.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
