subscribe

[Erik van oosten]

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Spark on secure HDFS

[Erik van oosten]

Hi,

We really would like to use Spark but we can’t because we have a secure HDFS 
environment (Cloudera).
I understood https://issues.apache.org/jira/browse/SPARK-2541 contains a patch. 
Can one of the committers please take a look?

Thanks!
 Erik.


—
Erik van Oosten
http://www.day-to-day-stuff.blogspot.com/




-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Opportunity to speed up toLocalIterator?

[Erik van Oosten]

Hi,

This might be an opportunity to give a huge speed bump to toLocalIterator.

Method toLocalIterator fetches the partitions to the driver one by one. 
This is great. What is not so great, is that any required computation 
for the yet-to-be-fetched-partitions is not kicked off until it is 
fetched. Effectively only one partition is being computed at the same 
time, giving idle resources and longer wait time.


Is this observation correct?

Is it possible to have concurrent computation on all partitions while 
retaining the download-a-partition at a time behavior?


Kind regards,
    Erik.

--
Erik van Oosten
http://www.day-to-day-stuff.blogspot.com/


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org