[jira] [Commented] (CSV-274) CSVParser.iterator() does not iterate over result set as expected.
[
https://issues.apache.org/jira/browse/CSV-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613153#comment-17613153
]
Peter Hull commented on CSV-274:
OK I've added PR https://github.com/apache/commons-csv/pull/270
> CSVParser.iterator() does not iterate over result set as expected.
> --
>
> Key: CSV-274
> URL: https://issues.apache.org/jira/browse/CSV-274
> Project: Commons CSV
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.8
>Reporter: David Guiney
>Priority: Major
>
> To return a stream of `CSVRecords` in a Spliterators, I need to call
> `CSVParser.getRecords().iterator()`. I worry that the `getRecords()` will
> load the records from the parser into memory, before creating the iterator
> which can be a problem with large CSV files.
> My code:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return
> StreamSupport.stream(Spliterators.spliteratorUnknownSize(parser.iterator(),
> 0), false);
> }
> }
> {code}
> and:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return StreamSupport.stream(parser.spliterator(), false);
> }
> }
> {code}
> When I collect the results of my method, it gives me
> {code:java}
> []
> {code}
>
> If I replace `parser.iterator()` with `parser.getRecords().iterator()` then I
> get the desired results. Is the iterator not meant to be an iterator of the
> list of `CSVRecord`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CSV-274) CSVParser.iterator() does not iterate over result set as expected.
[
https://issues.apache.org/jira/browse/CSV-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612966#comment-17612966
]
Gary D. Gregory commented on CSV-274:
-
Hi [~peterhull90] and all,
If you have any ideas on improving the documentation or code, please do feel
free to open a PR on GitHub :)
> CSVParser.iterator() does not iterate over result set as expected.
> --
>
> Key: CSV-274
> URL: https://issues.apache.org/jira/browse/CSV-274
> Project: Commons CSV
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.8
>Reporter: David Guiney
>Priority: Major
>
> To return a stream of `CSVRecords` in a Spliterators, I need to call
> `CSVParser.getRecords().iterator()`. I worry that the `getRecords()` will
> load the records from the parser into memory, before creating the iterator
> which can be a problem with large CSV files.
> My code:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return
> StreamSupport.stream(Spliterators.spliteratorUnknownSize(parser.iterator(),
> 0), false);
> }
> }
> {code}
> and:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return StreamSupport.stream(parser.spliterator(), false);
> }
> }
> {code}
> When I collect the results of my method, it gives me
> {code:java}
> []
> {code}
>
> If I replace `parser.iterator()` with `parser.getRecords().iterator()` then I
> get the desired results. Is the iterator not meant to be an iterator of the
> list of `CSVRecord`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CSV-274) CSVParser.iterator() does not iterate over result set as expected.
[
https://issues.apache.org/jira/browse/CSV-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612947#comment-17612947
]
Peter Hull commented on CSV-274:
I think this is not a bug that should be fixed in the code, but maybe this kind
of usage could be warned about in the docs?
As I understand it the issue is that the parser gets closed as control exits
the try-with-resources block, hence the iterator and the stream that wraps it
will not return any more records. I also made this mistake, and I felt it was
not immediately obvious why my code was wrong.
> CSVParser.iterator() does not iterate over result set as expected.
> --
>
> Key: CSV-274
> URL: https://issues.apache.org/jira/browse/CSV-274
> Project: Commons CSV
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.8
>Reporter: David Guiney
>Priority: Major
>
> To return a stream of `CSVRecords` in a Spliterators, I need to call
> `CSVParser.getRecords().iterator()`. I worry that the `getRecords()` will
> load the records from the parser into memory, before creating the iterator
> which can be a problem with large CSV files.
> My code:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return
> StreamSupport.stream(Spliterators.spliteratorUnknownSize(parser.iterator(),
> 0), false);
> }
> }
> {code}
> and:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return StreamSupport.stream(parser.spliterator(), false);
> }
> }
> {code}
> When I collect the results of my method, it gives me
> {code:java}
> []
> {code}
>
> If I replace `parser.iterator()` with `parser.getRecords().iterator()` then I
> get the desired results. Is the iterator not meant to be an iterator of the
> list of `CSVRecord`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CSV-274) CSVParser.iterator() does not iterate over result set as expected.
[
https://issues.apache.org/jira/browse/CSV-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504592#comment-17504592
]
Gary D. Gregory commented on CSV-274:
-
Hi [~dguiney4]
As you can read in the Javadoc for {{{}CSVParser.getRecords(){}}}:
{quote}Parses the CSV input according to the given format and returns the
content as a list of
CSVRecord CSVRecords.
{quote}
Obviously, the only way to do that is to read the whole input.
If you look at 1.9.0-SNAPSHOT
([https://repository.apache.org/content/repositories/snapshots]) or git master,
you'll find:
{code:java}
/**
* Returns a sequential {@code Stream} with this collection as its source.
*
* @return a sequential {@code Stream} with this collection as its source.
* @since 1.9.0
*/
public Stream stream() {
return
StreamSupport.stream(Spliterators.spliteratorUnknownSize(iterator(),
Spliterator.ORDERED), false);
}
{code}
Which works and is tested in {{{}CSVParserTest.testStream(){}}}.
> CSVParser.iterator() does not iterate over result set as expected.
> --
>
> Key: CSV-274
> URL: https://issues.apache.org/jira/browse/CSV-274
> Project: Commons CSV
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.8
>Reporter: David Guiney
>Priority: Major
>
> To return a stream of `CSVRecords` in a Spliterators, I need to call
> `CSVParser.getRecords().iterator()`. I worry that the `getRecords()` will
> load the records from the parser into memory, before creating the iterator
> which can be a problem with large CSV files.
> My code:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return
> StreamSupport.stream(Spliterators.spliteratorUnknownSize(parser.iterator(),
> 0), false);
> }
> }
> {code}
> and:
> {code:java}
> public Stream convertFileToMaps(Path path) throws IOException {
> try (CSVParser parser = CSVParser.parse(path,
> Charset.defaultCharset(), CSVFormat.RFC4180
> .withFirstRecordAsHeader())) {
> return StreamSupport.stream(parser.spliterator(), false);
> }
> }
> {code}
> When I collect the results of my method, it gives me
> {code:java}
> []
> {code}
>
> If I replace `parser.iterator()` with `parser.getRecords().iterator()` then I
> get the desired results. Is the iterator not meant to be an iterator of the
> list of `CSVRecord`.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
