For this you will need to contribute... Le 27 oct. 2016 1:35 PM, "Mich Talebzadeh" <mich.talebza...@gmail.com> a écrit :
> so I assume Ignite will not work with Spark version >=2? > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 27 October 2016 at 12:27, vincent gromakowski < > vincent.gromakow...@gmail.com> wrote: > >> some options: >> - ignite for spark 1.5, can deep store on cassandra >> - alluxio for all spark versions, can deep store on hdfs, gluster... >> >> ==> these are best for sharing between jobs >> >> - shared sparkcontext and fair scheduling, seems to be not thread safe >> - spark jobserver and namedRDD, CRUD thread safe RDD sharing between >> spark jobs >> ==> these are best for sharing between users >> >> 2016-10-27 12:59 GMT+02:00 vincent gromakowski < >> vincent.gromakow...@gmail.com>: >> >>> I would prefer sharing the spark context and using FAIR scheduler for >>> user concurrency >>> >>> Le 27 oct. 2016 12:48 PM, "Mich Talebzadeh" <mich.talebza...@gmail.com> >>> a écrit : >>> >>>> thanks Vince. >>>> >>>> So Ignite uses some hash/in-memory indexing. >>>> >>>> The question is in practice is there much use case to use these two >>>> fabrics for sharing RDDs. >>>> >>>> Remember all RDBMSs do this through shared memory. >>>> >>>> In layman's term if I have two independent spark-submit running, can >>>> they share result set. For example the same tempTable etc? >>>> >>>> Cheers >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 27 October 2016 at 11:44, vincent gromakowski < >>>> vincent.gromakow...@gmail.com> wrote: >>>> >>>>> Ignite works only with spark 1.5 >>>>> Ignite leverage indexes >>>>> Alluxio provides tiering >>>>> Alluxio easily integrates with underlying FS >>>>> >>>>> Le 27 oct. 2016 12:39 PM, "Mich Talebzadeh" <mich.talebza...@gmail.com> >>>>> a écrit : >>>>> >>>>>> Thanks Chanh, >>>>>> >>>>>> Can it share RDDs. >>>>>> >>>>>> Personally I have not used either Alluxio or Ignite. >>>>>> >>>>>> >>>>>> 1. Are there major differences between these two >>>>>> 2. Have you tried Alluxio for sharing Spark RDDs and if so do you >>>>>> have any experience you can kindly share >>>>>> >>>>>> Regards >>>>>> >>>>>> >>>>>> Dr Mich Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> LinkedIn * >>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>> >>>>>> >>>>>> >>>>>> http://talebzadehmich.wordpress.com >>>>>> >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> On 27 October 2016 at 11:29, Chanh Le <giaosu...@gmail.com> wrote: >>>>>> >>>>>>> Hi Mich, >>>>>>> Alluxio is the good option to go. >>>>>>> >>>>>>> Regards, >>>>>>> Chanh >>>>>>> >>>>>>> On Oct 27, 2016, at 5:28 PM, Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>> >>>>>>> There was a mention of using Zeppelin to share RDDs with many users. >>>>>>> From the notes on Zeppelin it appears that this is sharing UI and I am >>>>>>> not >>>>>>> sure how easy it is going to be changing the result set with different >>>>>>> users modifying say sql queries. >>>>>>> >>>>>>> There is also the idea of caching RDDs with something like Apache >>>>>>> Ignite. Has anyone really tried this. Will that work with multiple >>>>>>> applications? >>>>>>> >>>>>>> It looks feasible as RDDs are immutable and so are registered >>>>>>> tempTables etc. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> Dr Mich Talebzadeh >>>>>>> >>>>>>> >>>>>>> LinkedIn * >>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>> >>>>>>> >>>>>>> http://talebzadehmich.wordpress.com >>>>>>> >>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>> for any loss, damage or destruction of data or any other property which >>>>>>> may >>>>>>> arise from relying on this email's technical content is explicitly >>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>> damages >>>>>>> arising from such loss, damage or destruction. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >> >
