[
https://issues.apache.org/jira/browse/TAJO-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231824#comment-14231824
]
Hyunsik Choi commented on TAJO-1222:
------------------------------------
This patch enables plan-text based file formats like Json and CSV to be
tolerant against parsing errors.
I added a table property {{text.error-tolerance.max-num}} to allow users to set
the maximum number of
permissible parsing errors. You can set this table property as follows:
{code:sql}
CREATE EXTERNAL TABLE ... (
...
...
model TEXT,
screen_height INT,
os TEXT,
createdat TEXT,
"data usage" TEXT
) USING JSON WITH ('text.error-tolerance.max-num' = '0') PARTITION BY
COLUMN (year TEXT, month TEXT) LOCATION 'file:///.../locket';
{code}
As I mentioned above, {{text.error-tolerance.max-num}} means the maximum number
of permissible parsing errors. Additionally, it can have three different
behaviors according to the number.
* If {{text.error-tolerance.max-num}} is set to negative integer (i.e., < 0), a
running query will ignore all parsing errors.
* If {{text.error-tolerance.max-num}} is set to 0, a running query does not
allow any parsing error.
* If {{text.error-tolerance.max-num}} is set to some positive integer number
(i.e., > 0), a running query only ignores the given number of parsing errors in
each task.
By default, 'text.error-tolerance.max-num' will be set to 0 if this property is
not specified.
> DelimitedTextFile should be tolerant against parsing errors.
> ------------------------------------------------------------
>
> Key: TAJO-1222
> URL: https://issues.apache.org/jira/browse/TAJO-1222
> Project: Tajo
> Issue Type: Bug
> Components: storage
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.9.1
>
>
> DelimitedTextFile is a base class for plan-text file formats like CSV or
> JSON. In practice, due to various reasons, parsing errors are usual. But, the
> current implementation does not allow any parsing error. It is inconvenient
> in many cases.
> The objective of this issue is to enable DelimitedTextFile to tolerate
> parsing errors up to the number given by users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
