[
https://issues.apache.org/jira/browse/PIG-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539810#comment-13539810
]
Russell Jurney commented on PIG-3059:
-------------------------------------
I actually just reviewed this, think that's my first time doing code review :)
1) It would be cool if InputErrorTracker could print the error rate at the end
of the job. Not sure I saw this? This would be my only suggestion that couldn't
be pushed to another JIRA.
2) Adding the file the error came from would be helpful. I imagine knowing
error distribution would be really helpful.
3) If there is any way for InputErrorTracker to write the exact input record
out to an error file, that would be awesome.
> Global configurable minimum 'bad record' thresholds
> ---------------------------------------------------
>
> Key: PIG-3059
> URL: https://issues.apache.org/jira/browse/PIG-3059
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.11
> Reporter: Russell Jurney
> Assignee: Cheolsoo Park
> Fix For: 0.12
>
> Attachments: PIG-3059.patch, test_avro_files.tar.gz
>
>
> See PIG-2614.
> Pig dies when one record in a LOAD of a billion records fails to parse. This
> is almost certainly not the desired behavior. elephant-bird and some other
> storage UDFs have minimum thresholds in terms of percent and count that must
> be exceeded before a job will fail outright.
> We need these limits to be configurable for Pig, globally. I've come to
> realize what a major problem Pig's crashing on bad records is for new Pig
> users. I believe this feature can greatly improve Pig.
> An example of a config would look like:
> pig.storage.bad.record.threshold=0.01
> pig.storage.bad.record.min=100
> A thorough discussion of this issue is available here:
> http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
