[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997798#comment-16997798 ]
Samuel Shepard commented on SPARK-6235: --------------------------------------- [~irashid] I meant the former (task result > 2G) as best I understand the architecture. Is there a different Jira for the ML library, since it affects PCA, that would be more appropriate? Thanks for the suggestions. Spark is a beautiful system with a lot of kind effort put into it. Computational biology has huge feature spaces all over the place. The two could really work well together, I think. This issue feels like some sort of left over from 32-bit Java, cramping Spark's style. :( > Address various 2G limits > ------------------------- > > Key: SPARK-6235 > URL: https://issues.apache.org/jira/browse/SPARK-6235 > Project: Spark > Issue Type: Umbrella > Components: Shuffle, Spark Core > Reporter: Reynold Xin > Priority: Major > Fix For: 2.4.0 > > Attachments: SPARK-6235_Design_V0.02.pdf > > > An umbrella ticket to track the various 2G limit we have in Spark, due to the > use of byte arrays and ByteBuffers. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org