Hi Jeroen, > However, am I correct in assuming that all the filtering will be then performed on the driver (since the .gz files are not splittable), albeit in several threads?
Filtering will not happen on the driver, it'll happen on executors, since `spark.read.json(…).filter(…).write(…)` is a separate job. But you have to submit each job in a separate thread, because each thread will get locked until the corresponding job finishes, so that's why you have to use `parallel collections`, you could also just use Futures, but it's just easier to use a `ParArray`. Internally it will work this way: once one task finishes decompressing a file, many tasks will get scheduled (based on `spark.default.parallelism`), and the executor that decompressed the file will start processing lines using all available threads, and after some time additional executors may join (based on the locality levels), and then after filtering, you would have to repartition back to 1 partition, so you could write just one `.gzip` file. And for each file, there will be a separate job, but because they all run within one Spark Context, executors will stay with the job, and will work on all files simultaneously. See more about scheduling within one application: https://spark.apache.org/docs/2.2.0/job-scheduling.html# scheduling-within-an-application On Fri, Sep 29, 2017 at 12:58 PM, Jeroen Miller <bluedasya...@gmail.com> wrote: > On Thu, Sep 28, 2017 at 11:55 PM, Jeroen Miller <bluedasya...@gmail.com> > wrote: > > On Thu, Sep 28, 2017 at 9:16 PM, Vadim Semenov > > <vadim.seme...@datadoghq.com> wrote: > >> Instead of having one job, you can try processing each file in a > separate > >> job, but run multiple jobs in parallel within one SparkContext. > > Hello Vadim, > > Today was a bit busy and I did not have the time to play with your > idea. However, am I correct in assuming that all the filtering will be > then performed on the driver (since the .gz files are not splittable), > albeit in several threads? > > If this is correct, then I guess the proper way to tackle this task > would be to run without any executors, but using all the cores and > memory of the machine for the driver? > > I will keep you posted on my progress, > > Thanks, > > Jeroen >
