This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 1325d13 ARROW-9101: [Doc][C++] Document encoding expected for CSV data
1325d13 is described below
commit 1325d139b7301df40ec0b22a74afd675aebe0d94
Author: Antoine Pitrou <[email protected]>
AuthorDate: Fri Jun 12 11:38:31 2020 -0500
ARROW-9101: [Doc][C++] Document encoding expected for CSV data
Closes #7407 from pitrou/ARROW-9101-document-csv-encoding
Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Wes McKinney <[email protected]>
---
docs/source/cpp/csv.rst | 17 +++++++++++++++++
docs/source/python/csv.rst | 15 ++++++++++++---
python/pyarrow/_csv.pyx | 6 ++++++
3 files changed, 35 insertions(+), 3 deletions(-)
diff --git a/docs/source/cpp/csv.rst b/docs/source/cpp/csv.rst
index 5be5483..5eaf191 100644
--- a/docs/source/cpp/csv.rst
+++ b/docs/source/cpp/csv.rst
@@ -107,6 +107,8 @@ column. Type inference considers the following data types,
in order:
* Boolean
* Timestamp (with seconds unit)
* Float64
+* Dictionary<String> (if :member:`ConvertOptions::auto_dict_encode` is true)
+* Dictionary<Binary> (if :member:`ConvertOptions::auto_dict_encode` is true)
* String
* Binary
@@ -126,6 +128,15 @@ can be chosen from the following list:
Other data types do not support conversion from CSV values and will error out.
+Dictionary inference
+--------------------
+
+If type inference is enabled and :member:`ConvertOptions::auto_dict_encode`
+is true, the CSV reader first tries to convert string-like columns to a
+dictionary-encoded string-like array. It switches to a plain string-like
+array when the threshold in :member:`ConvertOptions::auto_dict_max_cardinality`
+is reached.
+
Nulls
-----
@@ -134,6 +145,12 @@ Null values are recognized from the spellings stored in
factory method will initialize a number of conventional null spellings such
as ``N/A``.
+Character encoding
+------------------
+
+CSV files are expected to be encoded in UTF8. However, non-UTF8 data
+is accepted for Binary columns.
+
Performance
===========
diff --git a/docs/source/python/csv.rst b/docs/source/python/csv.rst
index c0fd8df..27a95fe 100644
--- a/docs/source/python/csv.rst
+++ b/docs/source/python/csv.rst
@@ -82,18 +82,27 @@ For memory-constrained environments, it is also possible to
read a CSV file
one batch at a time, using :func:`open_csv`. It currently doesn't support
parallel reading.
+Character encoding
+------------------
+
+CSV files are expected to be encoded in UTF8. However, non-UTF8 data
+is accepted for ``binary`` columns.
+
Performance
-----------
Due to the structure of CSV files, one cannot expect the same levels of
performance as when reading dedicated binary formats like
:ref:`Parquet <Parquet>`. Nevertheless, Arrow strives to reduce the
-overhead of reading CSV files.
+overhead of reading CSV files. A reasonable expectation is at least
+100 MB/s per core on a modern desktop machine (measured in source CSV bytes,
+not target Arrow data bytes).
Performance options can be controlled through the :class:`ReadOptions` class.
Multi-threaded reading is the default for highest performance, distributing
the workload efficiently over all available cores.
.. note::
- The number of threads to use concurrently is automatically inferred by Arrow
- and can be inspected using the :func:`~pyarrow.cpu_count()` function.
+ The number of concurrent threads is automatically inferred by Arrow.
+ You can inspect and change it using the :func:`~pyarrow.cpu_count()`
+ and :func:`~pyarrow.set_cpu_count()` functions, respectively.
diff --git a/python/pyarrow/_csv.pyx b/python/pyarrow/_csv.pyx
index 950e7dd..da514ce 100644
--- a/python/pyarrow/_csv.pyx
+++ b/python/pyarrow/_csv.pyx
@@ -658,6 +658,9 @@ def read_csv(input_file, read_options=None,
parse_options=None,
"""
Read a Table from a stream of CSV data.
+ The input CSV data should be encoded in UTF8. Non-UTF8 data can still
+ be read and converted as Binary columns.
+
Parameters
----------
input_file: string, path or file-like object
@@ -709,6 +712,9 @@ def open_csv(input_file, read_options=None,
parse_options=None,
"""
Open a streaming reader of CSV data.
+ The input CSV data should be encoded in UTF8. Non-UTF8 data can still
+ be read and converted as Binary columns.
+
Reading using this function is always single-threaded.
Parameters
