[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995889#comment-16995889 ]
Samuel Shepard commented on SPARK-6235: --------------------------------------- One use case could be fetching large results to the driver when computing PCA on large square matrices (e.g., distance matrices, similar to Classical MDS). This is very helpful in bioinformatics. Sorry if this already fixed past 2.4.0... > Address various 2G limits > ------------------------- > > Key: SPARK-6235 > URL: https://issues.apache.org/jira/browse/SPARK-6235 > Project: Spark > Issue Type: Umbrella > Components: Shuffle, Spark Core > Reporter: Reynold Xin > Priority: Major > Fix For: 2.4.0 > > Attachments: SPARK-6235_Design_V0.02.pdf > > > An umbrella ticket to track the various 2G limit we have in Spark, due to the > use of byte arrays and ByteBuffers. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org