spark optimized pagination

2018-06-09 Thread onmstester onmstester
Hi, I'm using spark on top of cassandra as backend CRUD of a Restfull Application. Most of Rest API's retrieve huge amount of data from cassandra and doing a lot of aggregation on them in spark which take some seconds. Problem: sometimes the output result would be a big list which make clien

Re: Spark / Scala code not recognising the path?

2018-06-09 Thread Abhijeet Kumar
The situation is completely different than what you are thinking. Ok, thanks for your time. From now I'll figure this out myself. Thank you again! On Sat, 9 Jun 2018, 13:27 Jörn Franke, wrote: > Why don’t you write the final name from the start? > Ie save as the file it should be named. > > On 9

Re: Spark / Scala code not recognising the path?

2018-06-09 Thread Jörn Franke
Why don’t you write the final name from the start? Ie save as the file it should be named. > On 9. Jun 2018, at 09:44, Abhijeet Kumar wrote: > > I need to rename the file. I can write a separate program for this, I think. > > Thanks, > Abhijeet Kumar >> On 09-Jun-2018, at 1:10 PM, Jörn Franke

Re: Spark / Scala code not recognising the path?

2018-06-09 Thread Abhijeet Kumar
I need to rename the file. I can write a separate program for this, I think. Thanks, Abhijeet Kumar > On 09-Jun-2018, at 1:10 PM, Jörn Franke wrote: > > That would be an anti pattern and would lead to bad software. > Please don’t do it for the sake of the people that use your software. > What d

Re: Spark / Scala code not recognising the path?

2018-06-09 Thread Jörn Franke
That would be an anti pattern and would lead to bad software. Please don’t do it for the sake of the people that use your software. What do you exactly want to achieve with the information if the file exists or not? > On 9. Jun 2018, at 08:34, Abhijeet Kumar wrote: > > Can you please tell the e