DjvuLee created SPARK-5739:
------------------------------
Summary: Size exceeds Integer.MAX_VALUE in FileMap
Key: SPARK-5739
URL: https://issues.apache.org/jira/browse/SPARK-5739
Project: Spark
Issue Type: Bug
Affects Versions: 1.1.1
Environment: Spark1.1.1 on a cluster with 12 node. Every node with
128GB RAM, 24 Core. the data is just 40GB, and there is 48 parallel task on a
node.
Reporter: DjvuLee
I just run the kmeans algorithm using a random generate data,but occurred this
problem after some iteration. I try several time, and this problem is
reproduced.
Because the data is random generate, so I guess is there a bug ? Or if random
data can lead to such a scenario that the size is bigger than
Integer.MAX_VALUE, can we check the size before using the file map?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
