afedulov commented on a change in pull request #17598:
URL: https://github.com/apache/flink/pull/17598#discussion_r792672907
##########
File path:
flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java
##########
@@ -155,16 +169,34 @@
private static final class Reader<R, T> implements StreamFormat.Reader<T> {
private final MappingIterator<R> iterator;
private final Converter<R, T, Void> converter;
+ private final boolean ignoreParseErrors;
- public Reader(MappingIterator<R> iterator, Converter<R, T, Void>
converter) {
+ public Reader(
+ MappingIterator<R> iterator,
+ Converter<R, T, Void> converter,
+ boolean ignoreParseErrors) {
this.iterator = checkNotNull(iterator);
this.converter = checkNotNull(converter);
+ this.ignoreParseErrors = ignoreParseErrors;
}
@Nullable
@Override
public T read() throws IOException {
- return iterator.hasNext() ? converter.convert(iterator.next(),
null) : null;
+ while (true) {
Review comment:
An arbitrary number of lines can be malformed. Let's say this is a
single CSV file
```
a,b,c // header
1,2,3 // expected
1, // malformed
1,2,3,4 // malformed
5,6,7 // expected
```
Should produce [1,2,3] and [5,6,7] and ignore the rest. The number of
malformed elements is unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
