Dmitry Sysolyatin created SPARK-44037:
-----------------------------------------
Summary: Add maxCharsPerRow option for CSV datasource
Key: SPARK-44037
URL: https://issues.apache.org/jira/browse/SPARK-44037
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.4.0
Reporter: Dmitry Sysolyatin
CSV datasource supports maxColumns and maxCharsPerColumn options. But those two
option does not allow restrict row size properly.
For instance, if I want to limit the row size to be less than or equal to 100,
and I set maxColumns to 10 and maxCharsPerColumn to 10, then
# User can not read column with size > 10 even if row size <= 100
# User can not read more then 10 columns where each column < 5 chars even if
row size <= 100
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
