Re: Propagating pyspark errors to Livy
I'm currently getting the tracking url and giving it back to the calling program. Guess that's the best option then. Thanks Sent from my iPhone > On 11 Aug 2017, at 2:55 PM, Saisai Shao wrote: > > I think you should check Spark application log to see the details, it is hard > for Livy to get actual error from Spark. > >> On Fri, Aug 11, 2017 at 12:03 PM, Vivek wrote: >> Hi, >> >> Is there anyway to propagate errors from pyspark back to the calling program >> via Livy? >> Currently the Livy logs only tells me that the batch job has failed. How do >> I get the actual error on the spark side. >> >> Regards >> Vivek >> >> >> Sent from my iPhone >
Re: Multiple Livy instances and load balancing
Thanks Meisam. Is this code in github? Is there a PR for this? Regards Vivek Sent from my iPhone > On 27 Jul 2017, at 12:51 AM, Meisam Fathi wrote: > > Hi Vivek, > > We are running multiple instances of Livy on our clusters. Our users can > create and access jobs on any of the Livy instances. > > To answer your particular question > > 1. Is this feature available in the 0.3 release? No. The feature is not > available out of the box–we modified Livy to add this feature. > 2. How would I name/number the multiple instances I bring up? The load > balancer knows all the instances. Users only interact with the load balancer. > 3. How does one load balance and send requests across the multiple instances? > Livy instances do not know that requests come from a load balancer. Each > processes requests it receives and updates a shared "session store", which we > implemented on top of ZooKeeper. > 4. Does Livy have a heartbeat mechanism to understand which or how many > instances are up? No. There is a heartbeat mechanism in Livy, but we do not > using it to detect live/dead Livy instances. Each instance only reacts to the > updates to the ZooKeeper "session store", which is shared by all instances of > Livy. > > Thanks, > Meisam > >> On Wed, Jul 26, 2017 at 1:36 AM Vivek wrote: >> Ok.so if I start multiple instances how will I know which instance to send >> the request to? >> And multiple instances would then be only controlled by the port id? >> >> >> Sent from my iPhone >> >>> On 26 Jul 2017, at 4:04 PM, Saisai Shao wrote: >>> >>> Current Livy doesn't support the things you mentioned here. You can start >>> multiple Livy in the cluster, but each LivyServer is a standalone service >>> doesn't aware the existence of others. >>> On Wed, Jul 26, 2017 at 10:27 AM, Vivek wrote: Hi, We are now considering moving into a uat environment using Livy at my company. Has anyone implemented multiple Livy instances on a single cluster with load balancing? A few questions. 1. Is this feature available in the 0.3 release? 2. How would I name/number the multiple instances I bring up? 3. How does one load balance and send requests across the multiple instances? 4. Does Livy have a heartbeat mechanism to understand which or how many instances are up? Any answers would be appreciated. Regards Vivek Sent from my iPhone >>>
Re: Input file as an argument og a Spark code
I had a similar requirement. I used webhdfs to first copy the file across to hdfs before starting the spark job via Livy. Sent from my iPhone > On 25 Jul 2017, at 9:39 AM, Saisai Shao wrote: > > I think you have to make this csv file accessible from Spark cluster, putting > to HDFS is one possible solution. > >> On Tue, Jul 25, 2017 at 1:26 AM, Joaquín Silva wrote: >> Hello, >> >> >> >> I'm building a BASH program (using Curl) that should run a Spark code >> remotely using Livy. But one of the code argument is a CSV file, how can I >> make that spark reads this file?. The file is going to be in client side, >> not in the Spark cluster machines. >> >> >> >> Regards, >> >> >> >> Joaquín Silva >> >> >> >
Re: Livy in production
Hi Prabhu, Very nice presentation. Regards Vivek Sent from my iPhone > On 5 Jul 2017, at 11:11 PM, Prabhu Kasinathan wrote: > > In PayPal, we made Livy as the entry point for all Spark jobs on all > production, R&D and dev clusters. Per day, we are running 10k+ spark yarn > jobs, 50k+ interactive codes per day and millions of Livy requests. To > achieve this, we made lot of customization and enhancements as per our jobs > requirements. > > For more details, check this video: https://youtu.be/Oqq3m4RP2tE > > Thanks! > Prabhu > >> On Jul 5, 2017, at 1:12 AM, Jeff Zhang wrote: >> >> >> AFAIK, paypal use livy in some of their internal products. >> >> >> Mike Botazzo 于2017年7月5日周三 下午4:09写道： >>> I'm interested in a production environment too >>> >>> >>>> On Tuesday, July 4, 2017 at 9:29:30 AM UTC+2, Vivek Suvarna wrote: >>>> Hi, >>>> >>>> Has anyone used Livy in a production environment in their firms? >>>> Im looking to see if we can start using this in a prod env and wanted to >>>> know if anyone has already done so? >>>> Also how is the support from cloudera for running Livy in production? >>>> >>>> Regards >>>> Vivek >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Livy User" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to livy-user+unsubscr...@cloudera.org. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Livy User" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to livy-user+unsubscr...@cloudera.org. > -- > You received this message because you are subscribed to the Google Groups > "Livy User" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to livy-user+unsubscr...@cloudera.org.
