Hi All , If some one can help me understand as which portion of the code gets executed on Driver and which portion will be executed on executor from the below code it would be a great help
I have to load data from 10 Tables and then use that data in various manipulation and i am using SPARK SQL for that please let me know if below code will be executed on the driver or it will be executed in each executor node. And if i do a join on the data frame will it happen on executor or driver ? options.put("dbtable", "(select * from t_table1) as t_table1"); DataFrame t_gsubmember = sqlContext.read().format("jdbc").options(options).load(); t_table1.cache(); options.put("dbtable", "(select * from t_table2) as t_table2"); DataFrame t_sub = sqlContext.read().format("jdbc").options(options).load(); t_table2.cache(); options.put("dbtable", "(select * from t_table3) as t_table3"); DataFrame t_pi = sqlContext.read().format("jdbc").options(options).load(); t_table3.cache(); ....And So on Thanks