[
https://issues.apache.org/jira/browse/SPARK-44037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Sysolyatin updated SPARK-44037:
--------------------------------------
Description:
CSV datasource supports maxColumns and maxCharsPerColumn options. But those two
options do not allow limit row size properly.
For instance, if I want to limit the row size to be less than or equal to 100,
and I set maxColumns to 10 and maxCharsPerColumn to 10, then
# User can not read column with size > 10 even if row size <= 100
# User can not read more than 10 columns where each column < 5 chars even if
row size <= 100
I suggest to add additional option maxCharsPerRow
was:
CSV datasource supports maxColumns and maxCharsPerColumn options. But those two
options do not allow limit row size properly.
For instance, if I want to limit the row size to be less than or equal to 100,
and I set maxColumns to 10 and maxCharsPerColumn to 10, then
# User can not read column with size > 10 even if row size <= 100
# User can not read more then 10 columns where each column < 5 chars even if
row size <= 100
I suggest to add additional option maxCharsPerRow
> Add maxCharsPerRow option for CSV datasource
> --------------------------------------------
>
> Key: SPARK-44037
> URL: https://issues.apache.org/jira/browse/SPARK-44037
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Dmitry Sysolyatin
> Priority: Major
>
> CSV datasource supports maxColumns and maxCharsPerColumn options. But those
> two options do not allow limit row size properly.
> For instance, if I want to limit the row size to be less than or equal to
> 100, and I set maxColumns to 10 and maxCharsPerColumn to 10, then
> # User can not read column with size > 10 even if row size <= 100
> # User can not read more than 10 columns where each column < 5 chars even if
> row size <= 100
> I suggest to add additional option maxCharsPerRow
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
