Re: Unable to create a dataframe from json dstream using pyspark

2016-07-28 Thread Sunil Kumar Chinnamgari
unds. now my recordings are much more crisp. it is one of the lowest prices pop filters on amazon so might as well buy it, they honestly work the same despite their pricing,", "overall": 5.0, "summary": "good", "unixReviewTime": 1393545600, "reviewTime": "02 28, 2014"} I am absolute new comer to spark streaming and started working on pet projects by reading documentation. Any help and guidance is greatly appreciated. Best Regards,Sunil Kumar Chinnamgari

spark-ec2 scripts with spark-2.0.0-preview

2016-06-14 Thread Sunil Kumar
Hi, The spark-ec2 scripts are missing from spark-2.0.0-preview. Is there a workaround available ? I tried to change the ec2 scripts to accomodate spark-2.0.0...If I call the release spark-2.0.0-preview, then it barfs because the command line argument : --spark-version=spark-2.0.0-preview  gets

Re: JIRA SPARK-2984

2016-06-09 Thread Sunil Kumar
. Even if it is an intrinsic problem with s3 (and I'm not super sure since I'm just reading this on mobile) - it would maybe be a good thing for us to document. On Thursday, June 9, 2016, Sunil Kumar <parvat_2...@yahoo.com> wrote: | Holden Thanks for your prompt reply... Any suggestions on the next

Re: JIRA SPARK-2984

2016-06-09 Thread Sunil Kumar
Holden Thanks for your prompt reply... Any suggestions on the next step ? Does this call for a new spark jira ticket or is this an issue for s3? Thx

JIRA SPARK-2984

2016-06-09 Thread Sunil Kumar
Hi, I am running into SPARK-2984 while running my spark 1.6.1 jobs over yarn in AWS. I have tried with spark.speculation=false but still see the same failure with _temporary file missing for task_xxx...This ticket is in resolved state. How can this be reopened ? Is there a workaround ? thanks

connecting beeline to spark sql thrift server

2016-01-06 Thread Sunil Kumar
Hi, I have an AWS spark EMR cluster running with spark 1.5.2, hadoop 2.6 and hive 1.0.0I brought up the spark sql thriftserver on this cluster with spark.sql.hive.metastore version set to 1.0 When I try to connect to this thriftserver remotely using beeline packaged  with