[ https://issues.apache.org/jira/browse/SPARK-8510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-8510. --------------------------------- Resolution: Won't Fix I am resolving this per comments in https://github.com/apache/spark/pull/8384 and {quote} there is a clear indication that there is not support or interest in acting on it, then resolve as Won’t Fix {quote} in http://spark.apache.org/contributing.html as I assume it is a soft-yes for not supporting. Please reopen the JIRA and the PR too if anyone feels I misunderstood. > NumPy arrays and matrices as values in sequence files > ----------------------------------------------------- > > Key: SPARK-8510 > URL: https://issues.apache.org/jira/browse/SPARK-8510 > Project: Spark > Issue Type: Improvement > Components: PySpark > Reporter: Peter Aberline > Priority: Minor > > Using the DoubleArrayWritable as an example, I have added support for storing > NumPy arrays and matrices as elements of Sequence Files. > Each value element is a discrete matrix or array. This is useful where you > have many matrices that you don't want to join into a single Spark DataFrame > to store in a Parquet file. > There seems to be demand for this functionality: > http://mail-archives.us.apache.org/mod_mbox/spark-user/201506.mbox/%3CCAJQK-mg1PUCc_hkV=q3n-01ioq_pkwe1g-c39ximco3khqn...@mail.gmail.com%3E > I originally put this work in PR 6995, but closed it after suggestions from a > user to use NumPy's built in serialization. My second version is in PR 8384. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org