[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566230#comment-15566230 ] Matt Sun commented on CSV-196: -- Add to what I just said, the CSV parser also takes care of spaces (if any). So if we format the parser not to remove leading and trailing spaces during initialization, we will get info of the original data. Whether or not the field was encapsulated is the only missing info. > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566209#comment-15566209 ] Matt Sun commented on CSV-196: -- You're right, parser reads characters. But as downstream users, they can count bytes by characters read. As I said, as a integration with Hadoop API, we want to know the original data read by the parser. What I asked is not having parsing to count the bytes, but only to set a flag indicating whether the field was enclosed by quotes. > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556070#comment-15556070 ] Emmanuel Bourg commented on CSV-196: But the parser reads characters, not bytes. That doesn't seem very reliable, and I suspect there are other cases where the parser ignores input characters (such as spaces between a delimiter and a quote). Flagging the quoted fields is likely to be insufficient. > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1534#comment-1534 ] Matt Sun commented on CSV-196: -- yes, pretty much. The bytes read from the source file, including encapsulators, spaces, line breaks etc. > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554422#comment-15554422 ] Emmanuel Bourg commented on CSV-196: Do I understand correctly that you need this info only for counting the number of bytes read? > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519691#comment-15519691 ] Gary Gregory commented on CSV-196: -- The HEAD of Git master. Gary > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519443#comment-15519443 ] Matt Sun commented on CSV-196: -- Sure, I can do. Which version should the patch based on? > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CSV-196) Store the info of whether a field is enclosed by quotes
[ https://issues.apache.org/jira/browse/CSV-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518771#comment-15518771 ] Benedikt Ritter commented on CSV-196: - Hello [~mattsun], would you like to contribute a patch? BR, Benedikt > Store the info of whether a field is enclosed by quotes > --- > > Key: CSV-196 > URL: https://issues.apache.org/jira/browse/CSV-196 > Project: Commons CSV > Issue Type: Improvement > Components: Parser >Affects Versions: 1.4 >Reporter: Matt Sun > Labels: easyfix, features, patch > Fix For: Patch Needed > > Original Estimate: 48h > Remaining Estimate: 48h > > It will be good to have CSVParser class to store the info of whether a field > was enclosed by quotes in the original source file. > For example, for this data sample: > A, B, C > a1, "b1", c1 > CSVParser gives us record a1, b1, c1, which is helpful because it parsed > double quotes, but we also lost the information of original data at the same > time. We can't tell from the CSVRecord returned whether the original data is > enclosed by double quotes or not. > In our use case, we are integrating Apache Hadoop APIs with Commons CSV. CSV > is one kind of input of Hadoop Jobs, which should support splitting input > data. To accurately split a CSV file into pieces, we need to count the bytes > of data CSVParser actually read. CSVParser doesn't have accurate information > of whether a field was enclosed by quotes, neither does it store raw data of > the original source. Downstream users of commons CSVParser is not able to get > those info. > To suggest a fix: Extend the token/CSVRecord to have a boolean field > indicating whether the column was enclosed by quotes. While Lexer is doing > getNextToken, set the flag if a field is encapsulated and successfully parsed. > I find another issue reported with similar request, but it was marked as > resolved: [CSV91] > https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
