[
https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146585#comment-16146585
]
Shivaram Venkataraman commented on SPARK-21349:
-----------------------------------------------
I think this might be that we create a ParallelCollectionRDD for the statement
`(1 to (24*365*3))` -- The values are stored in the partition for this RDD [1]
[~dongjoon] If you use fewer values (i.e. say 1 to 100) or more partitions (I'm
not sure how many partitions are created in this example) does the warning go
away ?
[1]
https://github.com/apache/spark/blob/e47f48c737052564e92903de16ff16707fae32c3/core/src/main/scala/org/apache/spark/rdd/ParallelCollectionRDD.scala#L32
> Make TASK_SIZE_TO_WARN_KB configurable
> --------------------------------------
>
> Key: SPARK-21349
> URL: https://issues.apache.org/jira/browse/SPARK-21349
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.6.3, 2.2.0
> Reporter: Dongjoon Hyun
> Priority: Minor
>
> Since Spark 1.1.0, Spark emits warning when task size exceeds a threshold,
> SPARK-2185. Although this is just a warning message, this issue tries to make
> `TASK_SIZE_TO_WARN_KB` into a normal Spark configuration for advanced users.
> According to the Jenkins log, we also have 123 warnings even in our unit test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
