I have a small application like this acc = sc.accumulate(5)
def t_f(x,):
global acc
sleep(5)
acc += x
def f(x):
global acc
thread = Thread(target = t_f, args = (x,))
thread.start()
# thread.join() # without this it doesn't work
rdd = sc.parallelize([1,2,4,1])
rdd.foreach(f)
sleep(30)
print(acc.value)
assert acc.value == 13
The code doesn't work unless I uncomment the thread.join
Any idea why?
