[ https://issues.apache.org/jira/browse/SPARK-42373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688531#comment-17688531 ]
Ted Chester Jenks commented on SPARK-42373: ------------------------------------------- For the main use-case for this, [#39907|https://github.com/apache/spark/pull/39907], I have settled to define an ordering that doesn't become unclear with these method names. > Remove unused blank line removal from CSVExprUtils > -------------------------------------------------- > > Key: SPARK-42373 > URL: https://issues.apache.org/jira/browse/SPARK-42373 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.3.1 > Reporter: Willi Raschkowski > Priority: Minor > > The non-multiline CSV read codepath contains references to removal of blank > lines throughout. This is not necessary as blank lines are removed by the > parser. Furthermore, it causes confusion, indicating that blank lines are > removed at this point when instead they are already omitted from the data. > The multiline code-path does not explicitly remove blank lines leading to > what looks like disparity in behavior between the two. > The codepath for {{DataFrameReader.csv(dataset: Dataset[String])}} does need > to explicitly skip lines, and this should be respected in {{CSVUtils}}. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org