Re: (python) Spark .textFile(s3://…) access denied 403 with valid credentials

2017-03-07 Thread Amjad ALSHABANI
Hi Jonhy, What is the master you are using with spark-submit? I ve had this problem before because Spark (different from CLI and boto3) was running in Yarn distributed mode (--master yarn) So the keys were not copied to all the executors' nodes so I have had to submit my spark job as

Spark Streaming problem with Yarn

2017-02-28 Thread Amjad ALSHABANI
Hi everyone, I m experiencing a problem with my spark streaming job when running it on yarn. The problem appears only when running this application in a Yarn queue along with other Tez/MR applications This problem is in processing time, which exceeds 1 minute for batches of 1 second. Normally

Loading file into executor classpath

2016-02-22 Thread Amjad ALSHABANI
Hello everybody, I ve implemented a Loganalyzer program in spark, which takes the logs from an apache log file and translate it to a given object, The regex of the log file is GROK, so I m using GROK library to extract the desired field When running the application locally, it succeded without

Re: spark-shell with Yarn failed

2015-07-19 Thread Amjad ALSHABANI
Sent from my iPhone On Jul 18, 2015, at 11:46 AM, Amjad ALSHABANI ashshab...@gmail.com wrote: Does anybody have any idea about the error I m having.. I am really clueless... And appreciate any idea :) Thanks in advance Amjad On Jul 17, 2015 5:37 PM, Amjad ALSHABANI ashshab...@gmail.com

Re: spark-shell with Yarn failed

2015-07-18 Thread Amjad ALSHABANI
Does anybody have any idea about the error I m having.. I am really clueless... And appreciate any idea :) Thanks in advance Amjad On Jul 17, 2015 5:37 PM, Amjad ALSHABANI ashshab...@gmail.com wrote: Hello, First of all I m a newbie in Spark , I m trying to start the spark-shell with yarn

spark-shell with Yarn failed

2015-07-17 Thread Amjad ALSHABANI
Hello, First of all I m a newbie in Spark , I m trying to start the spark-shell with yarn cluster by running: $ spark-shell --master yarn-client Sometimes it goes well, but most of the time I got an error: Container exited with a non-zero exit code 10 Failing this attempt. Failing the