some options: - ignite for spark 1.5, can deep store on cassandra - alluxio for all spark versions, can deep store on hdfs, gluster...
==> these are best for sharing between jobs - shared sparkcontext and fair scheduling, seems to be not thread safe - spark jobserver and namedRDD, CRUD thread safe RDD sharing between spark jobs ==> these are best for sharing between users 2016-10-27 12:59 GMT+02:00 vincent gromakowski < vincent.gromakow...@gmail.com>: > I would prefer sharing the spark context and using FAIR scheduler for > user concurrency > > Le 27 oct. 2016 12:48 PM, "Mich Talebzadeh" <mich.talebza...@gmail.com> a > écrit : > >> thanks Vince. >> >> So Ignite uses some hash/in-memory indexing. >> >> The question is in practice is there much use case to use these two >> fabrics for sharing RDDs. >> >> Remember all RDBMSs do this through shared memory. >> >> In layman's term if I have two independent spark-submit running, can they >> share result set. For example the same tempTable etc? >> >> Cheers >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 27 October 2016 at 11:44, vincent gromakowski < >> vincent.gromakow...@gmail.com> wrote: >> >>> Ignite works only with spark 1.5 >>> Ignite leverage indexes >>> Alluxio provides tiering >>> Alluxio easily integrates with underlying FS >>> >>> Le 27 oct. 2016 12:39 PM, "Mich Talebzadeh" <mich.talebza...@gmail.com> >>> a écrit : >>> >>>> Thanks Chanh, >>>> >>>> Can it share RDDs. >>>> >>>> Personally I have not used either Alluxio or Ignite. >>>> >>>> >>>> 1. Are there major differences between these two >>>> 2. Have you tried Alluxio for sharing Spark RDDs and if so do you >>>> have any experience you can kindly share >>>> >>>> Regards >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 27 October 2016 at 11:29, Chanh Le <giaosu...@gmail.com> wrote: >>>> >>>>> Hi Mich, >>>>> Alluxio is the good option to go. >>>>> >>>>> Regards, >>>>> Chanh >>>>> >>>>> On Oct 27, 2016, at 5:28 PM, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>> >>>>> There was a mention of using Zeppelin to share RDDs with many users. >>>>> From the notes on Zeppelin it appears that this is sharing UI and I am not >>>>> sure how easy it is going to be changing the result set with different >>>>> users modifying say sql queries. >>>>> >>>>> There is also the idea of caching RDDs with something like Apache >>>>> Ignite. Has anyone really tried this. Will that work with multiple >>>>> applications? >>>>> >>>>> It looks feasible as RDDs are immutable and so are registered >>>>> tempTables etc. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>
