Re: SVD in pyspark ?
Hi Andreas, There unfortunately is not a Python API yet for distributed matrices or their operations. Here's the JIRA to follow to stay up-to-date on it: https://issues.apache.org/jira/browse/SPARK-3956 There are internal wrappers (used to create the Python API), but they are not really public APIs. The bigger challenge is creating/storing the distributed matrix in Python. Joseph On Sun, Jan 25, 2015 at 11:32 AM, Chip Senkbeil chip.senkb...@gmail.com wrote: Hi Andreas, With regard to the notebook interface, you can use the Spark Kernel ( https://github.com/ibm-et/spark-kernel) as the backend for an IPython 3.0 notebook. The kernel is designed to be the foundation for interactive applications connecting to Apache Spark and uses the IPython 5.0 message protocol - used by IPython 3.0 - to communicate. See the getting started section here: https://github.com/ibm-et/spark-kernel/wiki/Getting-Started-with-the-Spark-Kernel It discusses getting IPython connected to a Spark Kernel. If you have any more questions, feel free to ask! Signed, Chip Senkbeil IBM Emerging Technologies Software Engineer On Sun Jan 25 2015 at 1:12:32 PM Andreas Rhode m.a.rh...@gmail.com wrote: Is the distributed SVD functionality exposed to Python yet? Seems it's only available to scala or java, unless I am missing something, looking for a pyspark equivalent to org.apache.spark.mllib.linalg.SingularValueDecomposition In case it's not there yet, is there a way to make a wrapper to call from python into the corresponding java/scala code? The reason for using python instead of just directly scala is that I like to take advantage of the notebook interface for visualization. As a side, is there a inotebook like interface for the scala based REPL? Thanks Andreas -- View this message in context: http://apache-spark-user-list. 1001560.n3.nabble.com/SVD-in-pyspark-tp21356.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
SVD in pyspark ?
Is the distributed SVD functionality exposed to Python yet? Seems it's only available to scala or java, unless I am missing something, looking for a pyspark equivalent to org.apache.spark.mllib.linalg.SingularValueDecomposition In case it's not there yet, is there a way to make a wrapper to call from python into the corresponding java/scala code? The reason for using python instead of just directly scala is that I like to take advantage of the notebook interface for visualization. As a side, is there a inotebook like interface for the scala based REPL? Thanks Andreas -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SVD-in-pyspark-tp21356.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: SVD in pyspark ?
Hi Andreas, With regard to the notebook interface, you can use the Spark Kernel ( https://github.com/ibm-et/spark-kernel) as the backend for an IPython 3.0 notebook. The kernel is designed to be the foundation for interactive applications connecting to Apache Spark and uses the IPython 5.0 message protocol - used by IPython 3.0 - to communicate. See the getting started section here: https://github.com/ibm-et/spark-kernel/wiki/Getting-Started-with-the-Spark-Kernel It discusses getting IPython connected to a Spark Kernel. If you have any more questions, feel free to ask! Signed, Chip Senkbeil IBM Emerging Technologies Software Engineer On Sun Jan 25 2015 at 1:12:32 PM Andreas Rhode m.a.rh...@gmail.com wrote: Is the distributed SVD functionality exposed to Python yet? Seems it's only available to scala or java, unless I am missing something, looking for a pyspark equivalent to org.apache.spark.mllib.linalg.SingularValueDecomposition In case it's not there yet, is there a way to make a wrapper to call from python into the corresponding java/scala code? The reason for using python instead of just directly scala is that I like to take advantage of the notebook interface for visualization. As a side, is there a inotebook like interface for the scala based REPL? Thanks Andreas -- View this message in context: http://apache-spark-user-list. 1001560.n3.nabble.com/SVD-in-pyspark-tp21356.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
