[
https://issues.apache.org/jira/browse/SPARK-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-5739:
---------------------------
Summary: Size exceeds Integer.MAX_VALUE in File Map (was: Size exceeds
Integer.MAX_VALUE in FileMap)
> Size exceeds Integer.MAX_VALUE in File Map
> ------------------------------------------
>
> Key: SPARK-5739
> URL: https://issues.apache.org/jira/browse/SPARK-5739
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Spark1.1.1 on a cluster with 12 node. Every node with
> 128GB RAM, 24 Core. the data is just 40GB, and there is 48 parallel task on a
> node.
> Reporter: DjvuLee
>
> I just run the kmeans algorithm using a random generate data,but occurred
> this problem after some iteration. I try several time, and this problem is
> reproduced.
> Because the data is random generate, so I guess is there a bug ? Or if random
> data can lead to such a scenario that the size is bigger than
> Integer.MAX_VALUE, can we check the size before using the file map?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
