> Can you elaborate on how to implement "shared sparkcontext and fair > scheduling" option?
It just reuse 1 Spark Context by not letting it stop when the application had done. Should check: livy, spark-jobserver FAIR https://spark.apache.org/docs/1.2.0/job-scheduling.html <https://spark.apache.org/docs/1.2.0/job-scheduling.html> just how you scheduler your job in the pool but FAIR help you run job in parallel vs FIFO (default) 1 job at the time. > My approach was to use sparkSession.getOrCreate() method and register temp > table in one application. However, I was not able to access this tempTable in > another application. Store metadata in Hive may help but I am not sure about this. I use Spark Thrift Server create table on that then let Zeppelin query from that. Regards, Chanh > On Oct 27, 2016, at 9:01 PM, Victor Shafran <victor.shaf...@equalum.io> wrote: > > Hi Vincent, > Can you elaborate on how to implement "shared sparkcontext and fair > scheduling" option? > > My approach was to use sparkSession.getOrCreate() method and register temp > table in one application. However, I was not able to access this tempTable in > another application. > You help is highly appreciated > Victor > > On Thu, Oct 27, 2016 at 4:31 PM, Gene Pang <gene.p...@gmail.com > <mailto:gene.p...@gmail.com>> wrote: > Hi Mich, > > Yes, Alluxio is commonly used to cache and share Spark RDDs and DataFrames > among different applications and contexts. The data typically stays in > memory, but with Alluxio's tiered storage, the "colder" data can be evicted > out to other medium, like SSDs and HDDs. Here is a blog post discussing Spark > RDDs and Alluxio: > https://www.alluxio.com/blog/effective-spark-rdds-with-alluxio > <https://www.alluxio.com/blog/effective-spark-rdds-with-alluxio> > > Also, Alluxio also has the concept of an "Under filesystem", which can help > you access your existing data across different storage systems. Here is more > information about the unified namespace abilities: > http://www.alluxio.org/docs/master/en/Unified-and-Transparent-Namespace.html > <http://www.alluxio.org/docs/master/en/Unified-and-Transparent-Namespace.html> > > Hope that helps, > Gene > > On Thu, Oct 27, 2016 at 3:39 AM, Mich Talebzadeh <mich.talebza...@gmail.com > <mailto:mich.talebza...@gmail.com>> wrote: > Thanks Chanh, > > Can it share RDDs. > > Personally I have not used either Alluxio or Ignite. > > Are there major differences between these two > Have you tried Alluxio for sharing Spark RDDs and if so do you have any > experience you can kindly share > Regards > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw> > > http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> > > Disclaimer: Use it at your own risk. Any and all responsibility for any loss, > damage or destruction of data or any other property which may arise from > relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > On 27 October 2016 at 11:29, Chanh Le <giaosu...@gmail.com > <mailto:giaosu...@gmail.com>> wrote: > Hi Mich, > Alluxio is the good option to go. > > Regards, > Chanh > >> On Oct 27, 2016, at 5:28 PM, Mich Talebzadeh <mich.talebza...@gmail.com >> <mailto:mich.talebza...@gmail.com>> wrote: >> >> >> There was a mention of using Zeppelin to share RDDs with many users. From >> the notes on Zeppelin it appears that this is sharing UI and I am not sure >> how easy it is going to be changing the result set with different users >> modifying say sql queries. >> >> There is also the idea of caching RDDs with something like Apache Ignite. >> Has anyone really tried this. Will that work with multiple applications? >> >> It looks feasible as RDDs are immutable and so are registered tempTables etc. >> >> Thanks >> >> >> Dr Mich Talebzadeh >> >> LinkedIn >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw> >> >> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> >> >> Disclaimer: Use it at your own risk. Any and all responsibility for any >> loss, damage or destruction of data or any other property which may arise >> from relying on this email's technical content is explicitly disclaimed. The >> author will in no case be liable for any monetary damages arising from such >> loss, damage or destruction. >> > > > > > > > -- > Victor Shafran > > VP R&D| Equalum > > > Mobile: +972-523854883 <tel:%2B972-523854883> | Email: > victor.shaf...@equalum.io <mailto:victor.shaf...@equalum.io>
