Any response on this one? Thanks in advance!
On Thu, Oct 5, 2017 at 1:44 PM, Tarun Kumar <[email protected]> wrote:
> Hi, I registered an accumulator in driver via
> sparkContext.register(myCustomAccumulator,
> "accumulator-name"). But this accumulator is not available in
> task.metrics.accumulators()
> list. Accumulator is not visible in spark UI as well.
>
> Does spark need different configuration to make accumulator visible to
> tasks?
>
> My task tries to access accumulator via following two ways, it doesn't
> seem to find the accumulator:
>
> First way:
>
> AccumulatorContext.lookForAccumulatorByName("accumulator-name").
>
> map(accum => {
> accum.asInstanceOf[MyCustomAccumulator].add(*k, v*))
> })
>
>
> Second way:
>
> taskContext.taskMetrics().accumulators().
> filter(_.name == Some("accumulator-name")).
>
> map(accum => {
> accum.asInstanceOf[MyCustomAccumulator].add(*k, v*))
> })
>
>
> Thanks
>
> Tarun
>
>
