Re: Livy with Spark package
Thanks and agreed :) On Thu, Aug 24, 2017 at 12:01 PM, Saisai Shaowrote: > You could set "spark.jars.packages" in `conf` field of session post API ( > https://github.com/apache/incubator-livy/blob/master/ > docs/rest-api.md#post-sessions). This is equal to --package in > spark-submit. > > BTW you'd better ask livy question in u...@livy.incubator.apache.org. > > Thanks > Jerry > > On Thu, Aug 24, 2017 at 8:11 AM, ayan guha wrote: > >> Hi >> >> I have a python program which I invoke as >> >> spark-submit --packages com.databricks:spark-avro_2.11:3.2.0 >> somefile.py "2017-08-23 02:00:00" and it works >> >> Now I want to submit this file using Livy. I could work out most of the >> stuff (like putting files to HDFS etc) but not able to understand how/where >> to configure the "packages" switch...Any help? >> -- >> Best Regards, >> Ayan Guha >> > > -- Best Regards, Ayan Guha
Re: Livy with Spark package
You could set "spark.jars.packages" in `conf` field of session post API ( https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md#post-sessions). This is equal to --package in spark-submit. BTW you'd better ask livy question in u...@livy.incubator.apache.org. Thanks Jerry On Thu, Aug 24, 2017 at 8:11 AM, ayan guhawrote: > Hi > > I have a python program which I invoke as > > spark-submit --packages com.databricks:spark-avro_2.11:3.2.0 somefile.py > "2017-08-23 02:00:00" and it works > > Now I want to submit this file using Livy. I could work out most of the > stuff (like putting files to HDFS etc) but not able to understand how/where > to configure the "packages" switch...Any help? > -- > Best Regards, > Ayan Guha >
Livy with Spark package
Hi I have a python program which I invoke as spark-submit --packages com.databricks:spark-avro_2.11:3.2.0 somefile.py "2017-08-23 02:00:00" and it works Now I want to submit this file using Livy. I could work out most of the stuff (like putting files to HDFS etc) but not able to understand how/where to configure the "packages" switch...Any help? -- Best Regards, Ayan Guha
Livy VS Spark Job Server
It will be helpful if someone can compare Livy and Spark Job Server. Thanks
Re: Livy with Spark
Hi Mich, 1. Each user could create a Livy session (batch or interactive), one session is backed by one Spark application, and the resource quota is the same as normal spark application (configured by spark.executor.cores/memory,. etc), and this will be passed to yarn if running on Yarn. This is basically no different compared to normal spark application. 2. Livy is just a Rest service, inside Livy you could create multiple sessions, each session is mapped to one spark application and resource managed under the cluster manager (like YARN). 3. One user could create one session, this session will be run as the user, it is similar to HiveServer's user impersonation or poxy user. On Wed, Dec 7, 2016 at 5:51 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks Richard. > > I saw your question in the above blog: > > How does Livy proxy the user? Per task? Do you know how quotas are > assigned to users, like how do you stop one Livy user from using all of the > resources available to the Executors? > > My points are: > > >1. Still don't understand how quotas are assigned to users. Is that >done by YARN? >2. What will happen if more than one Livy is running on the same >cluster all controlled by the same YARN. how resouces are allocated > > cheers > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 5 December 2016 at 14:50, Richard Startin <richardstar...@outlook.com> > wrote: > >> There is a great write up on Livy at >> http://henning.kropponline.de/2016/11/06/ >> >> On 5 Dec 2016, at 14:34, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Hi, >> >> Has there been any experience using Livy with Spark to share multiple >> Spark contexts? >> >> thanks >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >
Re: Livy with Spark
Thanks Richard. I saw your question in the above blog: How does Livy proxy the user? Per task? Do you know how quotas are assigned to users, like how do you stop one Livy user from using all of the resources available to the Executors? My points are: 1. Still don't understand how quotas are assigned to users. Is that done by YARN? 2. What will happen if more than one Livy is running on the same cluster all controlled by the same YARN. how resouces are allocated cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 5 December 2016 at 14:50, Richard Startin <richardstar...@outlook.com> wrote: > There is a great write up on Livy at > http://henning.kropponline.de/2016/11/06/ > > On 5 Dec 2016, at 14:34, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Hi, > > Has there been any experience using Livy with Spark to share multiple > Spark contexts? > > thanks > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > >
Re: Livy with Spark
You mean share a single spark context across multiple jobs? https://github.com/spark-jobserver/spark-jobserver does the same On Mon, Dec 5, 2016 at 9:33 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi, > > Has there been any experience using Livy with Spark to share multiple > Spark contexts? > > thanks > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > >
Re: Livy with Spark
Thanks Richard for the link. Also its interaction with Zeppelin is great. I believe it is a very early stage for now Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 5 December 2016 at 14:50, Richard Startin <richardstar...@outlook.com> wrote: > There is a great write up on Livy at > http://henning.kropponline.de/2016/11/06/ > > On 5 Dec 2016, at 14:34, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Hi, > > Has there been any experience using Livy with Spark to share multiple > Spark contexts? > > thanks > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > >
Re: Livy with Spark
There is a great write up on Livy at http://henning.kropponline.de/2016/11/06/ On 5 Dec 2016, at 14:34, Mich Talebzadeh <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>> wrote: Hi, Has there been any experience using Livy with Spark to share multiple Spark contexts? thanks Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
Livy with Spark
Hi, Has there been any experience using Livy with Spark to share multiple Spark contexts? thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
Re: LIVY VS Spark Job Server
I have experience with both Livy & spark-jobserver. spark-jobserver gives you better API, particularly, if you want to work within a single spark context. Livy supports submitting python & R code while spark-jobserver doesn't support it. spark-jobserver code is more complex, it actively uses Akka, and for me it was difficult to debug and add additional features to it. Livy's architecture is simpler in this sense. I tried to add `yarn-cluster` support to spark-jobserver but got lost halfway-through while Livy supports yarn-cluster mode out of the box and it was easy for me to add missing features to Livy. I'm not going into more details, but I'm ready to answer some specific questions about usage of both Livy & spark-jobserver. On Wed, Sep 14, 2016 at 3:32 PM, SamyaMaiti <samya.maiti2...@gmail.com> wrote: > Hi Team, > > I am evaluating different ways to submit & monitor spark Jobs using REST > Interfaces. > > When to use Livy vs Spark Job Server? > > Regards, > Sam > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/LIVY-VS-Spark-Job-Server-tp27722.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
LIVY VS Spark Job Server
Hi Team, I am evaluating different ways to submit & monitor spark Jobs using REST Interfaces. When to use Livy vs Spark Job Server? Regards, Sam -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/LIVY-VS-Spark-Job-Server-tp27722.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org