Long running Spark Job Status on Remote Submission

2017-11-20 Thread Harsh Choudhary
. How do I get the status of such long-running jobs so that I can do the further tasks on my remote machine after the job completion? Livy is one choice but I want to do it without that, if possible. *Thanks!* Harsh Choudhary

Re: Database insert happening two times

2017-10-17 Thread Harsh Choudhary
.are the multiple rows being written dupes (they have all same >> fields/values)? >> Hth >> >> >> On Oct 17, 2017 1:08 PM, "Harsh Choudhary" <shry.ha...@gmail.com> wrote: >> >>> This is the code - >>> hdfs_path= >>> if(hd

Re: Database insert happening two times

2017-10-17 Thread Harsh Choudhary
val updatelambdaReq:InvokeRequest = new InvokeRequest(); updatelambdaReq.setFunctionName(updateFunctionName); updatelambdaReq.setPayload(updatedLambdaJson.toString()); System.out.println("Calling lambda to add log"); val updateLambdaResult = byteBufferToString(lambdaClient.invoke(updat

Database insert happening two times

2017-10-17 Thread Harsh Choudhary
of code in workers? If it is so then how can I solve it so that it only writes once. *Thanks!* *Cheers!* Harsh Choudhary