I would prefer sharing the spark context and using FAIR scheduler for user concurrency
Le 27 oct. 2016 12:48 PM, "Mich Talebzadeh" <mich.talebza...@gmail.com> a écrit : > thanks Vince. > > So Ignite uses some hash/in-memory indexing. > > The question is in practice is there much use case to use these two > fabrics for sharing RDDs. > > Remember all RDBMSs do this through shared memory. > > In layman's term if I have two independent spark-submit running, can they > share result set. For example the same tempTable etc? > > Cheers > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 27 October 2016 at 11:44, vincent gromakowski < > vincent.gromakow...@gmail.com> wrote: > >> Ignite works only with spark 1.5 >> Ignite leverage indexes >> Alluxio provides tiering >> Alluxio easily integrates with underlying FS >> >> Le 27 oct. 2016 12:39 PM, "Mich Talebzadeh" <mich.talebza...@gmail.com> >> a écrit : >> >>> Thanks Chanh, >>> >>> Can it share RDDs. >>> >>> Personally I have not used either Alluxio or Ignite. >>> >>> >>> 1. Are there major differences between these two >>> 2. Have you tried Alluxio for sharing Spark RDDs and if so do you >>> have any experience you can kindly share >>> >>> Regards >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 27 October 2016 at 11:29, Chanh Le <giaosu...@gmail.com> wrote: >>> >>>> Hi Mich, >>>> Alluxio is the good option to go. >>>> >>>> Regards, >>>> Chanh >>>> >>>> On Oct 27, 2016, at 5:28 PM, Mich Talebzadeh <mich.talebza...@gmail.com> >>>> wrote: >>>> >>>> >>>> There was a mention of using Zeppelin to share RDDs with many users. >>>> From the notes on Zeppelin it appears that this is sharing UI and I am not >>>> sure how easy it is going to be changing the result set with different >>>> users modifying say sql queries. >>>> >>>> There is also the idea of caching RDDs with something like Apache >>>> Ignite. Has anyone really tried this. Will that work with multiple >>>> applications? >>>> >>>> It looks feasible as RDDs are immutable and so are registered >>>> tempTables etc. >>>> >>>> Thanks >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> >>> >
