Any potentiail issue if I create a SparkContext in executor

[Shuai Zheng]

Hi All,

 

I notice if we create a spark context in driver, we need to call stop method
to clear it.

 

              SparkConf sparkConf = new
SparkConf().setAppName("FinancialEngineExecutor");

              JavaSparkContext ctx = new JavaSparkContext(sparkConf);

.

              String inputPath =
props[0].getProperty(Constants.S3_INPUT_FILES);

              JavaRDD<String> lines = ctx.textFile(inputPath);

              EngineFlatMapFunction engine = new EngineFlatMapFunction();

              engine.setAnalysisConfiguraitons(props);

              

              lines.mapPartitionsToPair(engine);

.

              ctx.stop();

And if I I have below code in the closure (EngineFlatMapFunction.java)

 

       Configuration hadoopConfiguration = new Configuration(new
JavaSparkContext(new SparkConf()).hadoopConfiguration());

 

Any issue there? Because I need to have the Hadoop configuration in closure
but the Configuration class itself is not serializable, so I retrieve it
from the executor part.

 

Will it have any issue if I create the Spark context in the above code
without call stop on it?

 

Regards,

 

Shuai