Multiple Livy instances and load balancing

2017-07-25 Thread Vivek
Hi, We are now considering moving into a uat environment using Livy at my company. Has anyone implemented multiple Livy instances on a single cluster with load balancing? A few questions. 1. Is this feature available in the 0.3 release? 2. How would I name/number the multiple instances I brin

Re: Livy jobs keeps runing forever

2017-07-25 Thread Marcelo Vanzin
On Tue, Jul 25, 2017 at 8:35 AM, Joaquín Silva wrote: > 17/07/25 14:21:46 ERROR yarn.ApplicationMaster: User class threw exception: > java.lang.OutOfMemoryError: PermGen space > > So in order to solve this issue I increased the executor and driver memory: > "driverMemory":"15g","executorMemory":"1

Livy jobs keeps runing forever

2017-07-25 Thread Joaquín Silva
Hello, I'm new in Livy, and I have this problem. When I run a batch job like this: curl -X POST --data '{"file": "/user/spark/program.jar", "className": "my.program"}' -H "Content-Type: application/json" LIVY-HOST:8998/batches The program keeps runing for ever. Searching in the logs

RE: Input file as an argument og a Spark code

2017-07-25 Thread Joaquín Silva
Well I think I will build a REST API over Livy in order to import/export data into HDFS. Thanks all! Joaquín Silva | Pentagon Security & AKAINIX Av. Kennedy 4.700, Piso 10, Of. 1002, Edificio New Century, Vitacura | Código Postal (ZIP Code) 7561127 Cel: (56-9) 6304 2498 From: Vivek