SparkConf is not a singleton.
However, SparkContext in almost all cases are. So you can use
SparkContext.getOrCreate().getConf
On Wed, Mar 16, 2016 at 12:38 AM, Pete Robbins <robbin...@gmail.com> wrote:
> I'm writing a metrics sink and reporter to push metrics to Elasticsearch.
> An example format of a metric in JSON:
>
> {
> "timestamp": "2016-03-15T16:11:19.314+0000",
> "hostName": "10.192.0.87"
> "applicationName": "My application",
> "applicationId": "app-20160315093931-0003",
> "executorId": "17",
> "executor_threadpool_completeTasks": 20
> }
>
> For correlating the metrics I want the timestamp, hostname, applicationId,
> executorId and applicationName.
>
> Currently I am extracting the applicationId and executor Id from the
> metric name as MetricsSystem prepends these to the name. As the sink is
> instantiated without the SparkConf I can not determine the applicationName.
>
> Another proposed change in
> https://issues.apache.org/jira/browse/SPARK-10610 would also make me
> require access to the SparkConf to get the applicationId/executorId.
>
> So, Is the SparkConf a singleton and can there be a Utils method for
> accessing it? Instantiating a SparkConf myself will not pick up the appName
> etc as these are set via methods on the conf.
>
> I'm trying to write this without modifying any Spark code by just using a
> definition in the metrics properties to load my sink.
>
> Cheers,
>
