[ https://issues.apache.org/jira/browse/SPARK-34339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278452#comment-17278452 ]
Apache Spark commented on SPARK-34339: -------------------------------------- User 'HeartSaVioR' has created a pull request for this issue: https://github.com/apache/spark/pull/31464 > Expose the number of truncated paths in Utils.buildLocationMetadata() > --------------------------------------------------------------------- > > Key: SPARK-34339 > URL: https://issues.apache.org/jira/browse/SPARK-34339 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.2.0 > Reporter: Jungtaek Lim > Priority: Major > > SPARK-31793 introduces Utils.buildLocationMetadata() to reduce the length of > location metadata. It effectively reduced the memory usage as only a few > paths will be included (and it's controlled by threshold value), but there's > no indication of the fact for truncation. > If the first 2 of 5 paths are only fit to the threshold, > Utils.buildLocationMetadata() shows the first 2 paths, but there's no mention > that 3 paths are truncated. Even no mention that some paths have been > truncated; it shows the same with the output just first 2 paths were > presented. This could bring confusion. > We should allow more space (like 10+ chars) to represent the fact N paths are > truncated. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org