[ https://issues.apache.org/jira/browse/SPARK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346499#comment-14346499 ]
kishorekumar neelamegam commented on SPARK-4368: ------------------------------------------------ If no changes or needed in Spark, can it be documented to show how Ceph storage can be used instead of HDFS. I read the above link on GlusterFS, doesn't provide much info. May be we'll play with Ceph storage and study its workings. Any pointers will be greatly appreciated. We have a Ceph cluster, and the concepts relate to HDFS. (distributed replicated storage) on block level. http://docs.megam.io/v1.0/docs/megam_ceph_works We are thinking on testing storage using Ceph as VMs use Ceph. HDFS hogs more core/memory as its JVM, where ceph is native. This is more of unification of storage for a customer and reduces cores/memory. > Ceph integration? > ----------------- > > Key: SPARK-4368 > URL: https://issues.apache.org/jira/browse/SPARK-4368 > Project: Spark > Issue Type: Improvement > Components: Input/Output > Reporter: Serge Smertin > > There is a use-case of storing big number of relatively small BLOB objects > (2-20Mb), which has to have some ugly workarounds in HDFS environments. There > is a need to process those BLOBs close to data themselves, so that's why > MapReduce paradigm is good, as it guarantees data locality. > Ceph seems to be one of the systems that maintains both of the properties > (small files and data locality) - > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-July/032119.html. I > know already that Spark supports GlusterFS - > http://mail-archives.apache.org/mod_mbox/spark-user/201404.mbox/%3ccf657f2b.5b3a1%25ven...@yarcdata.com%3E > So i wonder, could there be an integration with this storage solution and > what could be the effort of doing that? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org