Re: Spark task hangs infinitely when accessing S3 from AWS

2016-01-27 Thread Gourav Sengupta
t; know the following: >>>> 1. Number of partitions >>>> 2. Number of files >>>> 3. Time taken to create the RDD's >>>> >>>> >>>> Regards, >>>> Gourav Sengupta >>>> >>>> >&g

Re: Spark task hangs infinitely when accessing S3 from AWS

2016-01-27 Thread Erisa Dervishi
;>> Gourav Sengupta >>> >>> >>> On Tue, Jan 26, 2016 at 1:12 PM, Gourav Sengupta < >>> gourav.sengu...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> are

Re: Spark task hangs infinitely when accessing S3 from AWS

2016-01-26 Thread Gourav Sengupta
. >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-task-hangs-infinitely-when-accessing-S3-from-AWS-tp25289p26068.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. &g

Re: Spark task hangs infinitely when accessing S3 from AWS

2016-01-26 Thread Erisa Dervishi
>> >>> Hi, >>> >>> are you creating RDD's out of the data? >>> >>> >>> >>> Regards, >>> Gourav >>> >>> On Tue, Jan 26, 2016 at 12:45 PM, aecc <alessandroa...@gmail.com> wrote: >>> >&g

Re: Spark task hangs infinitely when accessing S3 from AWS

2016-01-26 Thread Gourav Sengupta
in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-task-hangs-infinitely-when-accessing-S3-from-AWS-tp25289p26068.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > -

Re: Spark task hangs infinitely when accessing S3 from AWS

2016-01-26 Thread Erisa Dervishi
8. > The number of partitions used when reading data is 7315. > The maximum size of a file to read is 14G > The size of the folder is around: 270G > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-task-hangs-infinitely-when-acces

Re: Spark task hangs infinitely when accessing S3 from AWS

2016-01-26 Thread aecc
Sorry, I have not been able to solve the issue. I used speculation mode as workaround to this. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-task-hangs-infinitely-when-accessing-S3-from-AWS-tp25289p26068.html Sent from the Apache Spark User List

Re: Spark task hangs infinitely when accessing S3 from AWS

2015-11-12 Thread aecc
Any hints? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-task-hangs-infinitely-when-accessing-S3-from-AWS-tp25289p25365.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Spark task hangs infinitely when accessing S3 from AWS

2015-11-12 Thread Michael Cutler
Reading files directly from Amazon S3 can be frustrating especially if you're dealing with a large number of input files, could you please elaborate more on your use-case? Does the S3 bucket in question already contain a large number of files? The implementation of the * wildcard operator in S3

Re: Spark task hangs infinitely when accessing S3 from AWS

2015-11-12 Thread aecc
/Spark-task-hangs-infinitely-when-accessing-S3-from-AWS-tp25289p25367.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e

Re: Spark task hangs infinitely when accessing S3 from AWS

2015-11-09 Thread aecc
Any help on this? this is really blocking me and I don't find any feasible solution yet. Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-task-hangs-infinitely-when-accessing-S3-from-AWS-tp25289p25327.html Sent from the Apache Spark User List

Spark task hangs infinitely when accessing S3 from AWS

2015-11-05 Thread aecc
: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-task-hangs-infinitely-when-accessing-S3-from-AWS-tp25289.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe,

Re: Spark task hangs infinitely when accessing S3

2015-09-14 Thread Akhil Das
Are you sitting behind a proxy or something? Can you look more into the executor logs? I have a strange feeling that you are blowing the memory (and possibly hitting GC etc). Thanks Best Regards On Thu, Sep 10, 2015 at 10:05 PM, Mario Pastorelli < mario.pastore...@teralytics.ch> wrote: > Dear

Spark task hangs infinitely when accessing S3

2015-09-10 Thread Mario Pastorelli
Dear community, I am facing a problem accessing data on S3 via Spark. My current configuration is the following: - Spark 1.4.1 - Hadoop 2.7.1 - hadoop-aws-2.7.1 - mesos 0.22.1 I am accessing the data using the s3a protocol but it just hangs. The job runs through the whole data set but