Hi Davies,Thank you for pointing to spark streaming. I am confused about how to
return the result after running a function via a thread.I tried using Queue to
add the results to it and print it at the end.But here, I can see the results
after all threads are finished.How to get the result of
I think this is a general multiple-threading question, Queue is the
right direction to go.
Have you try something like this?
results = Queue.Queue()
def run_job(f, args):
r = f(*args)
results.put(r)
# start multiple threads to run jobs
threading.Thread(target=run_job, args=(f,
Hi Akhil, The python wrapper for Spark Job Server did not help me. I actually
need the pyspark code sample which shows how I can call a function from 2
threads and execute it simultaneously. Thanks Regards,
Meethu M
On Thursday, 14 May 2015 12:38 PM, Akhil Das
Hi
So to be clear, do you want to run one operation in multiple threads within
a function or you want run multiple jobs using multiple threads? I am
wondering why python thread module can't be used? Or you have already gave
it a try?
On 18 May 2015 16:39, MEETHU MATHEW meethu2...@yahoo.co.in
SparkContext can be used in multiple threads (Spark streaming works
with multiple threads), for example:
import threading
import time
def show(x):
time.sleep(1)
print x
def job():
sc.parallelize(range(100)).foreach(show)
threading.Thread(target=job).start()
On Mon, May 18,
Did you happened to have a look at the spark job server?
https://github.com/ooyala/spark-jobserver Someone wrote a python wrapper
https://github.com/wangqiang8511/spark_job_manager around it, give it a
try.
Thanks
Best Regards
On Thu, May 14, 2015 at 11:10 AM, MEETHU MATHEW
Hi all,
Quote Inside a given Spark application (SparkContext instance), multiple
parallel jobs can run simultaneously if they were submitted from separate
threads.
How to run multiple jobs in one SPARKCONTEXT using separate threads in pyspark?
I found some examples in scala and java, but
