[
https://issues.apache.org/jira/browse/ARROW-15878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504691#comment-17504691
]
Yibo Cai edited comment on ARROW-15878 at 3/11/22, 3:45 AM:
------------------------------------------------------------
20% improvement for best case (with more memory consumption) looks not very
useful in practice.
Note: for the best test case, every cell contains one quote. Real world dataset
may have much fewer cells containing quotes. So the 20% improvement for best
case is probably useless.
was (Author: yibo):
20% improvement for best case (with more memory consumption) looks not very
useful in practice.
> [C++] Optimize csv writer for string with quotes
> ------------------------------------------------
>
> Key: ARROW-15878
> URL: https://issues.apache.org/jira/browse/ARROW-15878
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Yibo Cai
> Assignee: Yibo Cai
> Priority: Major
> Attachments:
> 0001-ARROW-15878-improve-csv-writer-for-string-with-quote.patch, wip.patch
>
>
> Escaping a string with quotes (put an extra quote before a quote) is the
> hotspot of csv writer [1]. This can probably be improved, possible approaches:
> - Find the next quote with memchr, then memcpy blocks without quotes.
> - Check if there are quotes with simd in 8 bytes or 16 bytes, do memcpy if
> no, otherwise go slow path.
> Should make sure the method doesn't decrease performance too much for strings
> with many quotes. And should be similar or better performance for short
> strings, which is common case.
> [1]
> [https://github.com/apache/arrow/blob/master/cpp/src/arrow/csv/writer.cc#L139]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
