Hi,
On Fri, Nov 14, 2014 at 2:49 PM, Jianshi Huang jianshi.hu...@gmail.com
wrote:
Ok, then we need another trick.
let's have an *implicit lazy var connection/context* around our code. And
setup() will trigger the eval and initialization.
Due to lazy evaluation, I think having
I see. Agree that lazy eval is not suitable for proper setup and teardown.
We also abandoned it due to inherent incompatibility between implicit and
lazy. It was fun to come up this trick though.
Jianshi
On Tue, Nov 18, 2014 at 10:28 AM, Tobias Pfeiffer t...@preferred.jp wrote:
Hi,
On Fri,
HI, all
Is there setup and cleanup function as in hadoop mapreduce in spark which does
some initialization and cleanup work?
Best Regards,
Kevin.
If you’re looking for executor side setup and cleanup functions, there
ain’t any yet, but you can achieve the same semantics via
|RDD.mapPartitions|.
Please check the “setup() and cleanup” section of this blog from
Cloudera for details:
So can I write it like this?
rdd.mapPartition(i = setup(); i).map(...).mapPartition(i = cleanup(); i)
So I don't need to mess up the logic and still can use map, filter and
other transformations for RDD.
Jianshi
On Fri, Nov 14, 2014 at 12:20 PM, Cheng Lian lian.cs@gmail.com wrote:
If
If you’re just relying on the side effect of |setup()| and |cleanup()|
then I think this trick is OK and pretty cleaner.
But if |setup()| returns, say, a DB connection, then the |map(...)| part
and |cleanup()| can’t get the connection object.
On 11/14/14 1:20 PM, Jianshi Huang wrote:
So
Ok, then we need another trick.
let's have an *implicit lazy var connection/context* around our code. And
setup() will trigger the eval and initialization.
The implicit lazy val/var trick is actually invented by Kevin. :)
Jianshi
On Fri, Nov 14, 2014 at 1:41 PM, Cheng Lian
