GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/12173
[SPARK-13792][SQL] Limit logging of bad records in CSVRelation
## What changes were proposed in this pull request?
Currently in `PERMISSIVE` and `DROPMALFORMED` modes we log any record that
is going to be ignored. This can generate a lot of logs with large datasets.
This pr is to log the parts of malformed records and the number of subsequent
records for each partition.
This adds two options as follows;
```
sqlContext.read
.format("csv")
.option("mode", "COUNTMALFORMED")
.option("maxStoredMalformedPerPartition", 3)
.load("test.csv").show
```
A logging message is;
```
16/04/05 16:42:12 WARN CSVRelation: # of total malformed lines: 25
3 malformed lines extracted and listed as follows;
ab ccc ddd ddd
ab ccc ddd ddd
...
```
## How was this patch tested?
Manual tests done
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-13792
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12173.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12173
----
commit f4580802e62bed4ece480c37826750fc4a492916
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2016-04-04T09:48:33Z
Add MalformedLinesInfo for storing # of malformed lines
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
