Umesh, What kind of resource scheduler are you using ? Is it Spark's standalone service ? If yes, you can start 18 executors by changing the spark-default.conf and restarting your spark cluster (see configs here <https://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts>) and information on how to do it here <https://spark.apache.org/docs/latest/spark-standalone.html#executors-scheduling>. Find details on how to do it for other resources schedulers on Spark Deployment tab in the documentation.
Now, 1 executor + 18 cores = 18 executors + 1 core. Hence, you can get the same parallelism either way. Unit of parallelism in Spark = Task = 1 core Thousands of parquet files will be spread over multiples tasks with 18 of them running in parallel in your case since you have 18 cores at your disposal. (PS : The OS might also do some pipelining and context switching for a single core but that's not very relevant here) Hope this helps. Thanks, Nishith On Fri, Mar 8, 2019 at 12:23 PM Umesh Kacha <[email protected]> wrote: > Ok that seems like moving away from distributed to single processing I have > 18 cores per executor now if I dont use all the cores what's the point of > having distributed systems. Also I am just curious how will spark unit of > parallelism work here if we have just one core per executor if I have > thousands of parquet files it means few executors each with one core so at > a time few parquet files will be loaded in spark partitions/tasks. Please > correct me if I am wrong. Thanks. > > On Sat, Mar 9, 2019 at 1:46 AM nishith agarwal <[email protected]> > wrote: > > > Umesh, > > > > This issue still persists. Could you please use num-cores = 1 ? You can > > scale out using num-executors. > > > > -Nishith > > > > On Fri, Mar 8, 2019 at 12:06 PM Umesh Kacha <[email protected]> > wrote: > > > > > I think issue is this https://github.com/uber/hudi/issues/227 I get > the > > > same error and I tried to use multiple executor cores 4 and I am using > > > Spark 2.2.0. Is this issue fixed? > > > > > > > > > > > > On Fri, Mar 8, 2019 at 6:58 PM Vinoth Chandar <[email protected]> > wrote: > > > > > > > Could you please share the entire stack trace? > > > > > > > > On Fri, Mar 8, 2019 at 1:56 AM Umesh Kacha <[email protected]> > > > wrote: > > > > > > > > > Hi I am using Spark Shell to save spark dataframe as Hoodie dataset > > > using > > > > > bulk insert option inside Hoodie spark datasource. It seems to be > > > working > > > > > and trying to save but in the end it fails giving the following > > > exception > > > > > > > > > > Failed to initialize HoodieStorageWriter for path > > > > > /tmp/hoodie-test/2019/blabla.parquet > > > > > > > > > > > > > > >
