[
https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496868#comment-16496868
]
Andrew K Long edited comment on SPARK-24442 at 5/31/18 5:09 PM:
----------------------------------------------------------------
Hey Sean,
Thanks for commenting!
"There are already method arguments for truncation and max rows, so I don't
know if it's worth the complexity to alter defaults with yet another config
param."
While there exists parameters there's no easy default way of adjusting the
width. I have several tables where there's always 22 characters of data and 22
rows of data so to actually get the data from the console I have to always add
the parameters.
> df.show
vs
> df.show(30,false)
"The naming convention doesn't quite match other spark params too."
I'm totally open to a better naming convention. it didn't quite seem to fit
with most of the other parameters,
"I wonder if there is any way to detect the terminal width with any
reliability, even if not in all cases? like how commonly is COLUMNS set in a
shell?"
I did a bit of research on this. According to stack overflow there's no
reliable cross-platform way of doing this.
[https://stackoverflow.com/questions/1286461/can-i-find-the-console-width-with-java?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa|https://stackoverflow.com/questions/1286461/can-i-find-the-console-width-with-java?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa)]
There is a library, JLine2, that claims to be able todo this but this would
require adding a whole new dependency which seems overkill when an optional
parameter will do the job just fine.
was (Author: andrewklong):
Hey Sean,
Thanks for commenting!
"There are already method arguments for truncation and max rows, so I don't
know if it's worth the complexity to alter defaults with yet another config
param."
While there exists parameters there's no easy default way of adjusting the
width. I have several tables where there's always 22 characters of data and 22
rows of data so to actually get the data from the console I have to always add
the parameters.
> df.show
vs
> df.show(30,false)
"The naming convention doesn't quite match other spark params too."
I'm totally open to a better naming convention. it didn't quite seem to fit
with most of the other parameters,
"I wonder if there is any way to detect the terminal width with any
reliability, even if not in all cases? like how commonly is COLUMNS set in a
shell?"
I did a bit of research on this. According to stack overflow there's no
reliable cross-platform way of doing this.
([https://stackoverflow.com/questions/1286461/can-i-find-the-console-width-with-java?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa)]
There is a library, JLine2, that claims to be able todo this but this would
require adding a whole new dependency which seems overkill when an optional
parameter will do the job just fine.
> Add configuration parameter to adjust the numbers of records and the charters
> per row before truncation when a user runs.show()
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-24442
> URL: https://issues.apache.org/jira/browse/SPARK-24442
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 2.2.0, 2.3.0
> Reporter: Andrew K Long
> Priority: Minor
> Fix For: 2.4.0
>
> Attachments: spark-adjustable-display-size.diff
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Currently the number of characters displayed when a user runs the .show()
> function on a data frame is hard coded. The current default is too small when
> used with wider console widths. This fix will add two parameters.
>
> parameter: "spark.show.default.number.of.rows" default: "20"
> parameter: "spark.show.default.truncate.characters.per.column" default: "20"
>
> This change will be backwords compatible and will not break any existing
> functionality nor change the default display characteristics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
